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ABSTRACT 

This  document  is  a report  of  a case  study  of  the  use 
of  a system  contractor  source  'selection  process  by  a federal 
program.  It  is  intended  to  be  of  both  research  and  admin- 
istrative value.  Its  primary  research  value  is  as  a bench 
mark  for  future  case  studies  on  source  selection  techniques 
in  general  and  for  future  research  on  the  specific  source 
selection  process  reported  on  here.  Its  primary  administra- 
tive value  is  to  those  responsible  for  procuring  systems 
(and  other  complex  and  uncertain  products,  e.g.,  large  eval- 
uations) as  an  evaluation  of  the  outcome  of  the  use  of  the 
specific  technique  in  terms  of  the  issues  critical  to  effec- 
tive source  selection. 

The  specific  problem  for  which  the  technique  was  used 

was  the  selection  of  two  contractors  to  develop  an  ongoing 

capability  to  evaluate  the  results  (agency  and  commercial 

impacts)  of  experimental  modifications  in  procurement  pro- 

# 

cedures  by  selected  federal,  state,  and  local  government 
agencies.  These  evaluation  systems  were  to  be  implemented 
by  organizations  outside  of  the  developing  organization 
(The  Experimental  Technology  Incentives  Program  of  the  Na- 
tional Bureau  of  Standards). 
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The  primary  administrative  finding  is  that  the  federal 
program  was  able  to  adapt  successfully  the  technique,  which 
was  developed  and  used  to  procure  "hardware”  systems,  to 
meet  its  requirements  to  select  contractors  to  develop  these 
"software"  systems.  The  successes  and  failures  in  dealing 
with  numerous  critical  issues  are  detailed  in  Chapter  V. 
Section  V.4  provides  a summary  of  these  results  with  cross- 
references  to  the  related  discussions  in  the  text. 

The  primary  research  contribution  is  the  offering  of 
an  evaluation  structure  and  an  example  of  its  use  for  future 
case  studies  of  source  selection  techniques.  This  work 
also  provides  a departure  point  for  further  research  on 
the  specific  technique  evaluated  here.  The  prior  literature 
is  void  of  detailed  case  material.  This  document  provides 
a structure  for  comparative  evaluation  and  much  of  the  data 
which  was  used  for  this  one. 

The  method  followed  is  to:  1)  provide  background  in 

terms  of  the  problems  of  the  specific  procurement  (section 
1.2)  and  related  procurements  (section  II. 2,  II. 3)  and  back- 
ground on  the  source  selection  process  evaluated  here  (sec- 
tion II. 5);  2)  review  the  related  issues  of  source  selection 
raised  in  prior  literature  (section  II. 4);  3)  supply  a de- 
tailed chronology  of  the  use  source  selection  process  (Chap- 
ter IV);  and '4)  evaluate  the  results  of  using  the  process 
with  a structure  built  around  the  critical  issues  raised 


V 


by  participants  (drawn  from  the  Chronology)  and  by  the  prior 
literature  (Chapter  II). 

Implications  of  this  study  for  future  use  of  the  pro- 
cess and  for  future  related  research  are  presented  in  Chap- 
ter VI.  A background  document  on  the  source  selection  pro- 
cess, documents  from  the  RFP  and  the  actual  ratings  from 
the  use  of  the  process  which  is  evaluated  here  are  provided 
in  the  Appendix. 

The  author  is  an  employee  of  the  federal  program  which 
used  this  process.  He  has  been  involved  in  two  subsequent 
source  selections  using  the  technique.  The  report  of  this 
research  is  the  author's  doctoral  dissertation.  His  commit- 
tee chairman  is  the  developer  of  the  technique  being  eval- 
uated. A self-report  of  the  author's  biases  appears  in 
section  III. 5. 
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CHAPTER  I 
INTRODUCTION 

I • 1 The  General  Probler!!  of  Acouirlng  Evaluation  Systems 

In  recent  years  the  level  of  resources  devoted  by  gov- 
ernment to  program  evaluation  has  grown  to  a significant 
level.  This  activity  is  expected  to  continue  to  expand 
with  increasing  congressional  interest  in  program  oversight. 
Because  of  requirements  for  evaluation,  some  programs  have 
become  interested  in  developing  capabilities  to  evaluate 
their  programs  on  an  on-going  basis.  Some  of  these  desired 
capabilities  have  been  labeled  as  evaluation  systems. 

Much  of  the  federal  government's  evaluation  work  is 
done  under  contracts.  The  track  record  of  such  evaluation 
has  not  been  good.  The  products  of  this  work  have  been 
criticized  as  being  of  faulty  design,  poor  timing,  and  ir- 
relevance for  use  in  decision  making. 

There  has  been  a growing  recognition  of  the  importance 
of  the  selection  of  the  contractor  as  a determinant  of  sat- 
isfaction with  the  outcome  of  the  evaluation.  Profession- 
als in  the  evaluation  community  have  begun  to  focus  on  the 
role  of  the  source  selection  process  in  evaluation. 


I • 2 The  Specific  Problem  which  ETIP  Faced 
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1.2.1  ETIP 

ETIP  (the  Experimental  Technology  Incentives  Program), 
at  the  National  Bureau  of  Standards  (NBS),  was  established 
by  Presidential  decree  in  1972  to  find  methods  which  the 
government  could  use  to  stimulate  technological  innovation. 
As  the  program  developed  it  pursued  this  mandate  by  working 
with  government  agencies  to  experiment  with  modifications 
in  policies  and  procedures  which  were  hypothesized  to  have 
an  inhibiting  effect  on  innovation.  The  idea  was  to  work 
with  those  people  in  government  who  have  the  responsibility 
for  various  programs  in  order  to  design,  implement,  and 
evaluate  the  results  of  these  "policy  experiments." 

1.2.2  ETIP’s  System  Procurement  Problem 

In  1975  ETIP  decided  that  it  wanted  to  develop  some 
kind  of  ongoing  capability  to  evaluate  the  results  (agency 
and  commercial  impacts)  of  the  experiments  in  procurement 
practice  which  it  was  conducting  with  various  federal, 
state,  and  local  government  agencies.  This  capability  was 
to  be  developed  using  outside  contractors.  The  responsi- 
bility for  this  fell  on  the  newly  hired  head  of  evaluation 
for  ETIP. 

He  faced  several  problems  in  trying  to  decide  how  to 
contract  for  this  evaluation  capability.  These  included 
that: 

0 This  project  was  only  one  of  his  responsibil- 
ities and  at  the  time  he  had  no  staff  working 


for  him.  This  created  a resource  problem  if 
he  were  to  meet  the  deadline  he  was  given  of 
awarding  the  contract  by  the  end  of  the  fiscal 
year. 

o He  had  no  directly  related  prior  experience 
with  either  the  source  selection  process  or 
with  systems  management. 

o He  was  not  yet  trusted  by  other  members  of 
the  ETIP  staff. 

o There  was  a great  deal  of  uncertainty  over 
what  the  requirements  for  the  evaluation  cap- 
ability would  be,  including  who  would  own  it, 
what  specific  types  of  experiments  it  would 
be  required  to  evaluate,  and  how  many  experi- 
ments would  be  evaluated  during  the  development 
of  the  systems. 

o The  evaluation  industry  had  a poor  track  record 
for  getting  its  work  used  in  decision-making. 

It  had  no  track  record  at  developing  evaluation 
systems . 

o ETIP  had  little  visibility  in  the  evaluation 
community.  Its  prior  attempt  at  contracting 
for  evaluation  services  resulted  in  cancella- 
tion of  the  RFP  after  the  proposals  had  been 
evaluated . 

o ETIP  had  a high  profile  with  the  administration 
of  MBS  and  parts  of  the  Department  of  Commerce. 
The  source  selection  would  be  subject  to  close 
internal  scrutiny. 

The  manager  of  evaluation  decided  to  ask  Charles  W. 
N.  Thompson,  a special  employee  of  the  program,  for  help. 
Thompson  chose  to  use  a source  selection  process  which  he 
had  developed  and  used  to  procure  "hardware”  subsystems 
for  the  Air  Force.  Use  of  the  method,  called  Parametric 
Factor  Evaluation  (PFE),  represented  a significant  depar- 
ture from  the  way  evaluations  were  being  purchased  by  the 


government . 
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There  was  a great  deal  of  skepticism  in  ETIP  over 
whether  the  source  selection  would  be  effective.  People 
were  concerned  with  whether: 

o the  process  was  legal; 
o any  firms  would  respond; 

o the  proposals  which  were  received  could  be 
evaluated;  and 

o the  selection  would  stand. 

I . 3 The  Topic  of  this  Research 

This  research  is  a case  study  of  ETIP’s  use  of  the 
Parametric  Factor  Evaluation  Process  to  select  two  contrac- 
tors to  develop  the  evaluation  systems  which  it  desired. 

The  case  study  is  constructed  by  the  author  from: 

o background  on  the  PFE  process  from  both  histori- 
cal documents  and  personal  communications  with 
its  primary  developer; 

o participation  in  the  process  at  ETIP  and  access 
to  records  of  the  source  selection;  and 

o interview  and  questionnaire  information  from 
other  participants  in  the  source  selection. 

The  format  followed  in  presenting  the  case  is  to  provide 

the  reader  with  background  on  both  the  source  selection 

process  and  the  PFE  process,  to  present  a specific  detailed 

chronology  of  ETIP’s  use  of  the  PFE  process,  and  to  evaluate 

ETIP’s  experience  against  the  set  of  relevant  issues.  Fi- 
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nally,  some  broader  implications  suggested  by  this  research 
are  presented. 

1 . 4 Value  of  this  Research 

This  research  is  intended  to  be  of  value  both  for  re- 
searchers and  for  people  responsible  for  acquiring  evalua- 
tion systems  (or  other  types  of  systems).  Its  value  for 
researchers  is  as  a bench  mark,  in  terms  of  method  and  re- 
sults, for  those  interested  in  further  research  on  PFE  or 
comparative  research  on  other  source  selection  techniques. 
It  is  also  offered  as  an  example  of  a credible  method  for 
doing  exploratory  research.  Its  value  for  people  respon- 
sible for  acquiring  systems  is  as  a description  of  the  PFE 
process  which  answers  some  of  the  questions  which  someone 
considering  using  the  process  may  have.  It  provides  a de- 
tailed level  of  information  on  how  the  process  works,  what 
problems  it  will  and  will  not  solve,  and  how  those  who  are 
involved  may  feel  about  it. 

1 . 5 The  Organization  of  this  Document 

This  document  is  organized  to  progress  from  the  gener- 
al to  the  specific  and  back  to  the  general.  Chapter  II 
contains  general  background  and  information  on  relevant 
prior  states-of-the-art  in  both  program  evaluation  and  sys- 
tems procurement.  The  size  of  the  problems,  the  people 
working  in  the  topic  areas,  the  prior  research,  and  the 
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issues  which  are  raised  by  other  authors  are  identified. 
Chapter  II  also  contains  background  on  the  Parametric  Factor 
Evaluation  process.  History  of  the  process,  claims  for 
where  it  will  be  useful,  and  a description  of  its  character- 
istics are  provided.  Chapter  III  describes  the  central 
purposes  of  the  research.  It  contains  a description  of 
both  what  the  author  set  out  to  accomplish  and  the  approach 
which  was  taken.  A discussion  of  the  proposed  value  of 
the  research  and  a disclosure  of  the  biases  of  the  author 
are  also  given.  Chapter  IV  is  a chronology  of  ETIP’s  use 
of  the  PFE  process  for  its  source  selection.  It  reports 
what  occurred,  including  what  issues  were  raised  by  partici- 
pants in  the  process.  Chapter  V presents  the  evaluation 
of  the  outcome  of  ETIP’s  use  of  PFE  against  a set  of  issues 
either  raised  by  participants  or  contained  in  related  liter- 
ature. Section  V.4  presents  a summary  of  the  results  in 
terms  of  successes,  failures,  and  remaining  uncertainties. 

It  contains  cross-references  back  to  the  discussion  of  each 
issue  in  the  preceding  text.  Chapter  VI  reviews  the  impli- 
cations of  this  research  which  go  beyond  the  specific  case 
study.  In  it  the  author  comments  on  the  methodology  used 
for  the  research,  draws  lessons  from  comparison  of  the  re- 
sults of  this  use  of  PFE  with  other  prior  and  subsequent 
uses,  mentions  other  problems  for  which  parts  of  the  PFE 
process  may  present  a useful  solution,  and  indicates  pos- 
sible topics  for  additional  research  on  PFE. 


7 


Two  terms  which  are  used  quite  often  in  later  chapters 
are  "bidder"  and  "panel."  The  term  "bidder"  is  used  to 
refer  to  a firm  responding  to  a request  for  proposals. 
Technically,  the  term  "offeror"  refers  to  a firm  responding 
to  a negotiated  procurement.  However,  since  the  use  of 
the  term  "bidder"  is  common  in  the  literature  and  in  prac- 
tice (including  use  in  ETIP's  RFP),  the  author  chose  to 
use  it  throughout  this  document.  The  term  "panel"  is  used 
to  refer  to  the  group  of  people  who  meet  to  perform  the 
review  and  scoring  of  the  technical  proposals. 
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CHAPTER  II 
BACKGROUND 


11 . 1 Introduction 

This  chapter  reviews  background  to  provide  perspective 
for  the  case  study  which  follows.  Information  is  presented 
on  the  related  states-of-the-art  in  terms  of  issues  raised 
by  researchers  and  practitioners  of  evaluation  and  systems 
procurement.  A description  of  the  Parametric  Factor  Evalua- 
tion approach  to  source  selection  is  also  presented.  The 
review  of  the  literature  is  not  intended  to  be  comprehen- 
sive. It  represents  a modest  selective  sampling  of  much 
larger  literatures  in  both  evaluation  and  systems  procure- 
ment. The  intention  is  to  give  the  reader  an  understanding 
of  the  larger  contexts  within  which  ETIP’s  procurement  prob- 
lem resided.  The  description  of  PFE  is  included  to  provide 
a context  for  the  case  study  in  terms  of  the  history  and 
characteristics  of  the  process  which  ETIP  used. 

11 . 2 Evaluation  Systems 

II. 2.1  Concept  of  Evaluation  Systems 

Evaluation  and  evaluation  systems  have  no  agreed  upon 
conceptual  framework  (Tien,  1979).  There  are,  however, 
many  government  programs  doing  evaluation  work  (Comptroller 
General,  1976).  This  work  varies  in  scope  and  method. 
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Some  of  it  is  classified  as  developing  evaluation  systems 
(Comptroller  General,  1978,  1979).  Other  works,  which  are 
large  program  evaluations,  share  many  of  the  same  character- 
istics and  problems  as  that  which  is  directed  toward  develop- 
ing evaluation  systems.  It  is  not  the  purpose  here  to  ana- 
lyze these  similarities  and  differences  but  merely  to  indi- 
cate some  of  the  problems  which  have  been  claimed  to  exist. 

Evaluation  systems  may  be  useful  to  programs  which 
are  expected  to  have  a continuing  requirement  for  informa- 
tion related  to  impacts,  where  at  least  some  of  it  is  of 
a stable  and  predictable  nature.  They  are  primarily  soft- 
ware systems  rather  than  hardware.  They  may  often  be  large 
and  complex  and  take  a long  time  to  acquire  (Thompson,  1976a). 

In  development  they  can  require  a large  number  of  people 

0 

of  diverse  skills  and  interests  to  work  together  over  a 
long  period  of  time.  As  such  they  exhibit  problems  of  co- 
operation and  communication  and  turnover  in  people  who  sup- 
port or  oppose  the  systems  (Patton,  1977;  Thompson,  1976a). 

For  comparison,  they  are  probably  most  closely  related  to 
management  information  systems  and  logistics  systems. 

The  definition  of  evaluation  system  used  by  ETIP  was; 

EVALUATION  SYSTEM  - Refers  to  the  set  of  policies 
and  procedures  which  provides  the  basis  for  eval- 
uating the  (agency  and/or  commercial)  impacts 
of  a set  of  specific  procurement  or  related  experi- 
ments. The  form,  detail  and  completeness  of  the 
system  will  progressively  change.  In  Phase  One, 
it  may  begin  as  a preliminary  outline  within  which 
detailed  "single  thread"  designs  are  developed 
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for  the  early  evaluation  of  specific  procurement  experi- 
ments; later,  the  results  of  the  preliminary  systems 
analysis  and  the  "pilot  test"  will  provide  the  basis 
for  a preliminary  evaluation  systems  design.  In  Phase 
Two,  progressive  refinement  should  result  in_a  rela- 
tively complete  evaluation  system  which  can  be  tested 
as  a prototype,  in  part,  through  the  evaluation  of 
specific  experiments.  In  Phase  Three,  the  evaluation 
system  should  be  in  the  form  of  a stable  and  complete 
system  (Thompson,  1976,  p.  5). 

II. 2. 2 The  Level  of  Evaluation  in  the  Federal  Govern- 
ment 

The  level  of  resources  devoted  to  government  program 
evaluation  has  increased  significantly  in  recent  years  and 
it  appears  that  this  trend  will  continue.  It  has  been  esti- 
mated that  the  federal  government  alone  spent  $243  million 
on  executive  branch  program  evaluation  in  fiscal  year  1977 
(Granquist,  1977).  This  may  be  compared  to  an  estimate 
of  $20  million  spent  in  fiscal  year  1969  (Knezo,  1974).^ 

In  the  last  several  years  Congress  has  demonstrated 
increased  interest  in  expanding  and  exercising  its  oversight 
authority  (Comptroller  General,  1977;  Havens,  1977).  On 
October  11,  1978,  the  Senate  passed  an  oversight  reform 
proposal,  S.  2,  The  Sunset  Act  of  1978.  This  legislation, 
which  has  facetiously  been  referred  to  as  the  Evaluators 
Full  Employment  Act,  would  require  periodic  review  and  re- 
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Though  the  methods  used  in  counting  probably  differed, 
these  figures  do  give  an  indication  of  magnitude  and  trend. 
It  should  also  be  noted  that  the  $243  million  does  not  in- 
clude Congressional  evaluation. 


authorization  of  nearly  all  federal  programs.  Though  the 
House  of  Representatives  did  not  pass  a similar  bill  during 
the  95th  Congress,  there  is  some  probability  that  similar 
legislation  will  exist  in  the  near  future.  The  result  will 
be  an  even  greater  demand  for  program  evaluations. 

II. 2. 3 The  Evaluation  Community 

A community  of  professional  evaluators  has  grown  up 
in  response  to  the  government's  demand  for  their  services. 
These  people  reside  both  within  and  outside  of  the  govern- 
ment. Within  the  government  most  executive  branch  agencies 
have  high-level  organizational  units  specifically  devoted 
to  evaluation  and  many  have  evaluation  functions  built  into 
the  lower  levels.  The  people  who  work  in  these  units  either 
perform  in-house  evaluations  or  manage  cor;tracts  and  grants 
which  support  their  evaluation  programs.  Congressional 
arms  of  the  government  such  as  the  General  Accounting  Office, 
the  Congressional  Budget  Office,  and  the  Office  of  Technol- 
ogy Assessment  also  have  sizable  in-house  evaluation  func- 
tions. Outside  of  government,  professional  evaluators  exist 
in  both  universities  and  in  contract  research  organizations. 
These  people  do  evaluations  for  government  programs  on  con- 
tracts or  grants  and  also  do  research  on  methods  for  improved 
evaluation.  In  a sense  they  form  a research  industry  (Bider- 
man  & Sharp,  1972). 


Communication  networks  have  evolved  to  serve  the  needs 


of  this  professional  community.  They  have  their  own  profes- 
sional organization  (e.g.,  The  Evaluation  Research  Society; 
The  Council  for  Applied  Social  Research)  and  journals  (e.g., 
Evaluation  Quarterly;  Evaluation ) , and  contribute  to  discus- 
sions in  the  media  of  other  related  organizations  (e.g., 

The  Institute  of  Management  Sciences;  The  American  Psycho- 
logical Association).  University  programs  have  developed 
to  research  topics  related  to  evaluation  and  to  teach  stu- 
dents how  to  do  this  work. 

II. 2. 4 The  Pains  of  Adolescence 

The  proliferation  of  evaluation  in  the  government  has 
brought  with  it  a debate  among  practitioners  and  intended 
users  as  to  the  quality  of  the  resultant  products.  This 
debate  takes  place  in  the  media  and  at  the  conferences  of 
evaluators  and  program  administrators  and  at  Congressional 
hearings.  It  often  centers  on  the  fact  that  the  results 
of  many  evaluations  are  not  used  in  decision-making  about 
the  programs  which  they  review  ( Agarwala-Rogers , 1977;  Berli 
1977;  Chelimsky,  1977,  Vol.  II,  pp.  33-44;  Cohen  & Caret, 
1975;  Weiss,  1972,  pp.  10-11;  Wholey,  Scanlon,  Duffy,  Fuku- 
mato  & Vogt,  1970;  Williams  ,&  Evans,  1969). 

Those  who  critique  evaluations  raise  questions  of  the 
relevance,  methodological  appropriateness,  and  political 
astuteness  of  evaluators.  Many  of  the  claims  of  problems 
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with  evaluators  center  on  the  irrelevance  of  their  work 
to  the  management  of  the  program  being  evaluated.  It  is 
claimed  that  evaluators  often  have  little  understanding 
of  the  program  processes  or  the  systems  in  which  the  pro- 
grams operate  (Horst,  Nay,  Scanlon,  Wholey,  1974).  They 
frequently  put  a great  deal  of  emphasis  on  peripheral  issues 
which  are  of  little  interest  to  program  managers  (Chelimsky, 
1977,  Vol  II,  p.  U3;  Granquist,  1977;  Guttentag,  1973;  Havens, 
1977).  Aspects  of  programs  being  evaluated  are  not  put 
into  a context  in  which  they  address  specific  policy  issues 
(Cook,  1978;  Lynch,  1976).  Evaluations  are  also  criticized 
for  being  submitted  too  late  to  be  of  use  in  decision-making 
(Evans,  1977;  Havens,  1977;  Wholey  et  al.,  1970). 

In  the  area  of  methodology  it  is  claimed  that  faulty 
or  simplistic  designs  are  used  which  lead  to  false  or  mis- 
leading or  suspect  conclusions  (Horst,  et  al.,  1974). 

The  relevance  of  measurement  schemes  to  the  program  being 
evaluated  (Horst,  et  aL,  1974),  the  adequacy  of  data  (Havens, 
1977),  and  the  quality  of  analysis  are  all  questioned  (Havens, 
1977).  Attacks  of  methodology  are  frequently  used  to  under- 
mine confidence  in  the  results  of  evaluations  (Evans,  1977). 

Evaluators  are  criticized  for  their  lack  of  politi- 
cal astuteness  (Patton  et  al,  1977).  There  has  been  a ten- 
dency on  the  part  of  some  evaluators  to  work  in  isolation 
from  the  managers  of  the  programs  they  are  evaluating  and 


to  favor  surprise  findings  (Brands,  1977;  Patton,  et  al., 
1977).  Related  to  this  is  the  mis-direction  of  evaluation 
reports  to  higher  levels  in  the  organization  or  to  the  press 
rather  than  on-going  presentation  of  results  to  those  with 
direct  control  of  the  programs  (Patton,  et  al,  1977). 
Evaluation  reports  have  also  been  criticized  for  their  large 
volume,  pretentiousness,  and  frequent  use  of  jargon  (Cook, 
1978;  Lynch,  1976). 

Evaluators  and  program  managers  may  work  toward  differ- 
ing purposes,  evaluators  being  concerned  with  technical 
excellence  of  research  designs  and  conclusiveness  of  attri- 
bution of  effect  and  program  people  being  concerned  with 
relevance,  timeliness,  breadth  of  analysis  and  responsive- 
ness (Chelimsky,  1977,  Vol.  II,  p.  31). 

II. 2. 5 Contract  Evaluation 

In  looking  for  causes  of  the  disappointing  outcomes 
of  evaluations,  some  authors  have  turned  attention  toward 
the  procurement  process  (Bernstein  & Freeman,  1975;  Biderman 
& Sharp,  1972,  1972a,  1974,  1974a;  Mitchell,  1973;  0MB, 
1975).  There  seems  to  be  a growing  recognition  of  the  im- 
portance of  finding  ways  for  basing  the  selection  of  con- 
tractors upon  criteria  which  have  a stronger  relationship 
to  expected  performance.  Articles  on  how  to  write  better 
RFPs  (Weidman,  1977)  and  critiques  of  issued  RFPs  (Evans 
& Anderson,  1978;  Hess,  Floden,  Sproull,  & Conry,  1978) 


have  appeared  in  journals.  A recent  meeting  of  the  Evalua- 
tion Research  Society  (November  1978)  held  a session  on 
the  RFP  Award  Process.  Another  professional  organization 
began  sponsorship  of  an  annual  award  for  the  best  evaluation 
RFP  (Council  for  Applied  Social  Research,  Note  1). 

Some  of  the  problems  which  are  raised  with  the  procure- 
ment process  include  the  high  cost  to  firms  of  submitting 
proposals,  the  lack  of  guidance  for  self  selection,  the 
vagueness  of  RFPs,  and  the  suspicion  that  the  government 
has  a preferred  source  in  mind  in  advance  of  receiving  pro- 
posals (Biderman  i Sharp,  1972a).  Another  problem  is  that 
the  evaluation  which  is  undertaken  often  has  little  relation 
ship  to  the  evaluation  which  was  described  in  the  RFP  (Bider 
man  & Sharp,  1972a).  Some  authors  question  the  competence 
of  federal  employees  to  develop  RFPs  and  evaluate  proposals. 
They  suggest  more  input  into  the  process  by  the  academic 
community,  (Bernstein  & Freeman,  1975),  including  issuing 
RFPs  for  contracts  to  write  RFPs  (Hess,  et.  al,  1978). 

II. 2. 6 Relationship  to  Systems  Procurement 

Comparisons  can  be  drawn  between  the  problems  of  acquir 
ing  evaluation  systems  and  the  problems  of  acquiring  weapons 
systems  (Thompson,  1978a;  Yarmolinsky,  1976).  However, 
the  literature  on  source  selection  for  weapons  systems  is 
rich  by  comparison  to  that  of  evaluation  systems.  It  is 
a more  mature  area  of  study  to  which  the  government  and 
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research  community  have  devoted  considerably  more  attention. 
Therefore,  it  is  useful  to  examine  the  literature  on  systems 
procurement  to  better  understand  the  procurement  process 
and  to  identify  the  issues  which  may  also  be  important  for 
evaluation  systems  procurement.  That  is  the  purpose  of 
section  II. 3» 

II. 2. 7 Summary  of  Evaluation  Systems  Background 
This  section  has  reviewed  some  of  the  literature  rela- 
ted to  evaluation  systems.  Emphasis  was  placed  on  identi- 
fying the  scope  of  federal  evaluation,  some  of  the  problems 
identified  with  its  use,  and  especially  those  identified 
with  the  procurement  process.  The  next  section  reviews 
literature  on  systems  procurement. 

II . 3 Systems  Procurement 

II. 3.1  The  Concept  of  Systems 

The  word  ’’system"  has  been  used  to  refer  to  many  differ 
ent  things  (Ackoff,  1971).  For  purposes  of  this  research, 
a brief  indication  of  system  characteristics  is  probably 
more  useful  than  using  any  one  definition.  Rudelius  (1969), 
indicates  aspects  of  large  systems  which  make  them  unique 
by  summarizing  the  first  three  chapters  of  Peck  and  Scherer 
(1962,  pp.  3-64).  He  lists  these  as  being  the  presence 


of: 


o a single  buyer, 
o a single  seller, 

o high  technological  and  political  uncertainty, 
o high  cost, 
o high  complexity, 
o high  lead  time, 
o limited  production,  and 

o contract  awards  made  before  development  of 
a precise  system  definition  (Rudelius,  1969, 
p.  23). 

Thompson  (1978)  describes  some  dimensions  which  may  be  used 

to  differentiate  among  various  systems.  These  are: 

o Hardware  < — •>  Software 

(the  degree  to  which  the  system  can  be  de- 
scribed in  terms  of  specific  physical  items), 

o Many  — •>  Few 

(the  degree  to  which  the  system  will  be  pro- 
duced in  quantity), 

o Incremental  < — > Novel 

(the  degree  to  which  the  system  represents 
a change  from  the  "system"  it  replaces), 
and 

o Inhouse  < — •>  Isolated 

(the  degree  to  which  the  developer  of  the 
system  is  responsive  to  the  direction  and 
control  of  the  user). 

II. 3.2  Size  of  System  Procurement 

Systems  procurements  represent  a significant  element 
of  federal  government  expenditures.  The  Office  of  Manage- 
ment and  Budget  Circular  Number  A-109  states  that’ 

the  acquisition  of  major  systems  by  the  Federal 
Government  constitutes  one  of  the  most  crucial 
and  expensive  activities  performed  to  meet  nation- 
al needs.  Its  impact  is  on  technology,  on  the 


Nation’s  economic  and  fiscal  policies,  and  on 
the  accomplishment  of  Government  agency  missions 
in  such  fields  as  defense,  space,  energy  and  trans- 
portation (Office  of  Management  Budget,  1976, 
p . 1 ) . 

Several  years  ago  Hall  (1975,  p.  88)  estimated  that  the 
Department  of  Defense  had  over  150  major  and  minor  systems 
acquisition  programs.  He  placed  the  cost  of  these  programs 
at  over  $200  billion  for  development  plus  an  amount  equal' 
for  follow-on. 

Much  systems  work  is  done  by  private  firms  under  con- 
tract to  the  government.  Livingston  (1959),  in  an  early 
article  on  the  system  concept,  indicates  that 

The  philosophy  underlying  the  weapon  system  con- 
cept holds  that  authority  for  the  design,  develop- 
ment, and  production  of  an  entire  weapon  or  sup- 
port system. .. should  be  assigned  to  a single 
[original  in  italics]  management  organization  (p. 

83). 

The  primary  reason  for  this  is  the  great  complexity  of  devel 
oping  and  integrating  all  of  the  system  components.  Since 
the  military  services  found  that  they  did  not  possess  suf- 
ficient in-house  capability  to  manage  the  development  of 
their  programs  they  began  contracting  for  systems  management 
(Livingston,  1959). 

II. 3. 3 Research  Related  to  Systems  Procurements 
Research  on  procurement  has  been  sponsored  by  both 
government  and  industry.  Spagnola  (1978,  p.  397)  traces 
the  Department  of  Defense’s  formal  interest  in  procurement 
research  to  the  Defense  Procurement  Pricing  Conference  held 
in  Hershey,  Pennsylvania  in  November,  1977.  Since  then 


the  various  services  have  set  up  offices  to  conduct  procure- 
ment research.  The  Army  established  its  Procurement  Research 
Office  (APRO)  in  1959;  the  Air  Force  established  the  Busi- 
ness Management  Research  Center  (AF3MRC)  in  1973;  and  the 
Navy  began  its  research  program  in  1977  (Spagnola,  1978, 
p.  397).  There  has  been  the  addition  of  a graduate  procure- 
ment curriculum  to  the  Air  Force  Institute  of  Technology 
and  a Systems  Acquisition  Management  curriculum  to  the  Naval 
Postgraduate  School  (Martin,  Heuer,  Kingston,  i Williams, 

1978,  p U22).  The  Defense  Systems  Management  College  has 
also  initiated  a program  of  acquisition  research  (Spagnola, 
1978,  p.  397).  In  1976  the  Office  of  Management  and  Budget 
established  the  Federal  Procurement  Institute  (now  the  Fed- 
eral Acquisition  Institute)  to  serve  as  a government-wide 
focal  point  for  procurement  research.  Industry  research 
has  been  sponsored  by  groups  such  as  the  Aerospace  Indus- 
tries Association  (e.g.,  Aerospace  Industries  Association, 
1969). 

The  research  in  this  area  is  reported  in  professional 
Journals  (e.g.,  Esposito,  1973;  Goldberg,  1977;  Skolnick, 
1969),  conference  proceedings  (e.g.,  Cochran  4 Rowe,  1977; 
Martin,  et.al,  1977),  theses  and  dissertations  (e.g.,  Bailey, 
1964;  Logan,  1966),  books  (e.g..  Fox,  1974;  Peck  4 Scherer, 
1962;  Scherer,  1964),  and  reports  (e.g..  Commission  on  Govern 
ment  Procurement,  1972).  It  is  conducted  both  by  practi- 
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tioners  (e.g.,  Carnes,  1976)  and  academics  (e.g.,  Hynes, 
1977;  Roberts,  1964). 

11.3.4  Characteristics  of  Systems  Procurements 

For  purposes  of  discussion,  the  procurement  of  systems 
is  divided  here  into  two  phases.  This  division  is  illustra- 
ted in  Figure  II. 1.  The  first  phase,  labeled  source  selec- 
tion, is  that  part  of  the  procurement  process  which  termin- 
ates with  the  signing  of  a contract  between  the  government 
and  a contractor.  The  second  phase,  labeled  the  system 
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N 
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* FIGURE  II. 1 TWO  PHASES  OF  THE  PROCUREMENT  OF  SYSTEMS 

development,  is  that  part  of  the  procurement  process  which 
follows  the  signing  of  the  contract  and  results  in  work 
toward  implementing  an  in-place  system.  Though  this  research 
is  primarily  concerned  with  the  source  selection  process, 
it  is  important  to  look  at  the  development  process  to  the 
extent  that  the  characteristics  of  this  process  interact 
with  source  selection.  That  is,  it  is  useful  to  look  at 
some  of  the  characteristics  of  the  development  process  which 
make  the  source  selection  process  difficult  and  in  turn 
some  of  the  characteristics  of  the  source  selection  process 
which  may  affect  the  development  process.  O’Connor  (1972) 
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has  pointed  out  that  early  misunderstandings  may  lead  to 
expensive  mistakes  downstream  in  the  program.  It  is  there- 
fore desirable  to  use  a source  selection  process  which  will 
be  able  to  predict  and  in  some  manner  favorably  affect  the 
development  process. 

The  characteristics  of  complexity  and  uncertainty  lead 
to  problems.  System  developments  have  both  technical  and 
organizational  components.  Technical  complexity  results 
from  the  large  number  of  technical  problems  present,  the 
interrelationship  among  these  problems,  and  reliability 
requirements  of  the  individual  components  (Peck  & Scherer 
1962,  p.  42).  Organizational  complexity  results  from  the 
large  number  of  objectives  which  different  people  have  for 
any  system,  the  changes  which  occur  in  objectives,  and  the 
need  to  resolve  conflicts  among  them  (Peck  & Scherer,  1962, 
p.  80).  Included  here  are  differences  between  the  objec- 
tives of  the  government  and  those  of  the  systems  contractor. 
Interface  between  technical  problems  and  organizational 
problems  is  important. 

Uncertainty  in  systems  development  results  from  the 
complexities  of  the  problem  and  the  required  long  develop- 
ment time  (Ruth,  1978;  Thompson,  1978).  Complexity  leads 
to  technical  difficulties  which  were  not  forseen  in  the 
early  stages  of  the  program  (Fox,  1974,  p.  372;  Peck  & Scher- 
er, 1962,  p.  24).  Some  of  these  problems  may  have  no  fea- 
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sible  solution  (Eyring,  1966,  p.  170)  and  must  result  in 
modifications  in  the  system's  performance  characteristics. 

The  long  development  time  may  result  in  required  changes 
due  to  advances  in  the  technical  state-of-the-art  (Peck 
& Scherer,  1962,  p.  46;  Srull,  1972,  p.  4),  modifications 
to  strategic  requirements  (Esposito,  1973,  P*  9;  Fox,  1974, 
p.  369;  Peck  & Scherer,  1962,  p.  48),  or  new  government 
policies  (Peck  & Scherer  1962,  p.  50).  The  demand  for  a 
system  may  also  vary  over  time,  particularly  as  competing 
systems  become  available  and  with  turnover  in  government 
personnel  who  support  or  oppose  them  (Thompson,  1978). 

There  will  also  be  turnover  of  personnel  by  the  system  con- 
tractor which  may  affect  approaches  used  (Ruth,  1978). 

The  literature  which  discusses  problems  with  the  devel- 
opment of  systems  emphasizes  the  large  numbers  of  changes 
which  occur.  These  changes  are  in  cost,  schedule,  and  per- 
formance characteristics.  Cost  is  often  discussed  in  terms 
of  overruns,  schedule  in  terms  of  slippages,  and  performance 
in  terms  of  both  increased  and  decreased  capabilities. 

Cost,  schedule,  and  performance  characteristics  are  all 
closely  interrelated  and  subject  to  similar  influences  (Ander- 
son, 1969,  p.  163;  Rudelius,  1969).  For  example  a technical 
problem  in  meeting  performance  capability  may  lead  to  in- 
creased cost  and  longer  time  until  delivery..  An  attempt 
to  cut  costs  may  lead  to  reduced  capabilities  and  longer 
time  until  delivery.  An  attempt  to  speed  delivery  may  lead 


to  greater  cost  and  reduced  performance.  Because  of  this 
interrelatedness,  the  causes  of  schedule  slippage  and  per- 
formance modifications  will  be  covered  here  within  the  dis- 
cussion of  cost  overruns  which  follows. 

The  literature  on  the  causes  of  overruns  can  be  grouped 

into  six  categories.  These  are; 

o having  to  lie  about  the  cost--buying  in, 

o spending  time  on  things  which  are  unimportant, 

o having  to  do  something  different  from  what 
was  originally  planned, 

0 doing  a poor  Job  of  estimating  costs, 

o making  mistakes  requiring  re-work,  and 

o catastrophic  external  events  (Cochran  & Rowe, 

1977;  Thompson,  Note  2). 

Buylng-ln  can  be  attributed  to  circumstances  which 
require  that  both  the  government  and  the  bidder  lie  about 
the  initial  estimated  cost  of  development.  The  government 
program  manager  may  use  an  unrealistically  optimistic  cost 
estimate  in  order  to  obtain  approval  for  his  system  (Fox, 
1974,  p.  159;  Peck  4 Scherer,  1962,  pp.  412-413).  Thus 
the  program  is  initially  inadequately  funded  (Patterson, 

1977,  p.  61).  Bidders  are  forced  by  the  government  to  under- 
estimate their  own  costs  if  they  wish  to  obtain  contracts 
(Peck  4 Scherer,  1962,  p.  412-413).  Bidders  often  believe 
that  there  is  little  chance  of  obtaining  future  contracts 
without  initial  contracts  (Babione,  1978;  Patterson,  1977, 
p.  60).  They  may  also  be  more  concerned  with  obtaining 


revenue  to  cover  fixed  costs  than  with  longer  term  cost 
issues  (Patterson,  1977,  p.  60).  Firms  realize  that  price 
may  be  the  primary  consideration  in  making  an  award  (Packard, 
1972),  since  the  government  must  defend  a selection  of  any- 
one other  than  the  lowest  bidder  (Fox,  1974,  p.  277),  and 
the  technical  evaluation  may  result  in  an  inability  to  dis- 
criminate among  bidders  (Carnes,  1976,  p.  2).  When  under 
contract  it  becomes  clear  that  the  award  price  will  not 
result  in  the  desired  performance  characteristics,  the  suc- 
cessful bidder  requests  contract  modifications  (Anderson, 
1969,  p.  163)  and  the  government  program  manager  requests 
a supplemental  budget  (Peck  & Scherer,  1962,  pp.  412-413). 

In  procurements  which  are  characterized  by  high  uncertainty, 
the  initial  price  may  have  little  relationship  to  the  cost 
to  perform  (Thompson,  1978a). 

Spending  time  on  things  which  are  unimportant  may  be 
related  to  differences  in  objectives  between  the  government 
and  its  contractor,  management  inefficiencies,  or  the  desire 
by  the  government  or  its  contractor  to  justify  future  costs. 
Objectives  of  the  contractor  which  are  not  held  by  the  gov- 
ernment may  lead  to  the  addition  to  systems  of  non-essential 
performance  characteristics  ( "goldplating" ) (Fox,  1974,  p. 
369).  Contractors  may  make  their  decisions  on  allocation 
of  effort  (and  on  doing  work  in-house  vs.  contracting  out) 
based  on  factors  such  as  desire  to: 


0 enhance  their  competitive  position  (e.g., 
maintain  competence,  public  image,  and  market 
share)  (Fox,  197^,  d.  441;  Peck  & Scherer, 

1962,  p.  458) 

o cover  overhead  (e.g.,  increased  direct  cost 
results  in  increased  charges  to  overhead 
and  indirect  cost)(Fox  1974,  pp.  232,  299), 
and 

o maintain  stability  of  employment  (e.g.,  during 
delays  while  waiting  for  approval  (Fox,  1974, 
pp.  165,  419;  Peck  & Scherer,  1962,  p.  458K 

Management  inefficiencies  may  be  the  result  of  lack  of  re- 
sponsibility for  managing  costs  (Fox,  1974,  p.  434),  or 
the  tendency  for  work  to  be  authorized  within  functional 
organizations  without  coordination  with  upper  level  manage- 
ment (Fox,  1974,  p.  419).  The  government  project  manager 
may  encourage  ’’goldplating"  in  order  to  justify  future  year 
budget  requests  (Fox,  1974,  p.  482).  By  increasing  costs 
the  contractor  may  also  try  to  justify  higher  future  esti- 
mates (Fox,  1974,  p.  232). 

Being  required  to  do  something  different  from  what 
was  originally  planned  results  from  the  need  to  respond 
to  the  customer’s  evolving  concept  of  uses  of  the  system 
(Cochran  & Rowe,  1977,  p.  540).  Changes  in  the  state-of- 
the-art  can  lead  to  required  new  features  which  were  not 
originally  anticipated  (Cochran  & Rowe,  1977;  Fox,  1974, 
p.  165).  In  many  instances  the  government  requires  bidders 
to  specify  approaches  in  their  proposals  in  areas  of  high 
uncertainty  (Drake,  1970,  p.  124).  This  can  lead  to  early 
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decisions  which  lock  the  program  into  a single  solution 
and  require  considerable  additional  expense  to  undertake 
other  options  (Hall,  1975,  p.  97;  Keeling,  1975,  p.  130). 

Poor  cost  estimates  are  attributed  to  not  dealing  ade- 
quately with  uncertainties.  Bidders  are  required  to  cost 
out  systems  with  little  or  no  historical  data  upon  which 
to  base  many  of  their  estimates.  This  is  particularly  true 
where  the  system  is  required  to  push  the  state-of-the-art 
(Fox,  1974,  p.  165;  Keeling,  1974,  p.  130;  Martin,  Glover 
& Lenz,  1977;  Srull,  1972,  p.  3).  The  estimates  which  result 
do  not  reflect  differences  in  uncertainty  over  the  costs 
of  various  parts  of  the  system  (Anderson,  1969;  Drake,  1970; 
Thompson,  1978a) . 

Mistakes  requiring  re-work  may  be  caused  by  setting 
overly  ambitious  requirements  or  by  poor  workmanship.  If 
the  original  requirements  call  for  significant  increases 
in  the  state-of-the-art  for  various  parts  of  the  system, 
it  is  possible  that  one  or  more  of  these  advances  will  prove 
too  ambitious  and  require  re-design  (Comptroller  General, 
1973,  p.  26).  Instances  of  poor  workmanship  may  also  re- 
quire re-work  (Fox,  1974,  p.  165). 

Catastrophic  external  events  may  lead  to  significant  ^ 
uncontrollable  overruns.  Included  here  are  events  such 
as  natural  disasters,  civil  disorder,  major  strikes,  and 
fire  (Cochran  & Rowe,  1977,  p.  536).  Severe  unanticipated 
inflation  might  also  be  included  in  this  category. 


Some  of  the  factors  which  lead  to  contract  changes 
may  be  anticipated  before  or  during  source  selection.  In 
some  cases  the  source  selection  process  may  be  a cause  of 
the  problem.  In  others  the  ability  to  be  able  to  better 
evaluate  the  technical  and  management  capabilities  of  a 
bidder  may  prevent  problems  from  occurring.  Some  feel  that 
the  selection  of  the  contractor  may  be  the  most  important 
aspect  of  an  acquisition  program  (Helman  & Taylor,  1976, 

p . 88  ) . 

11.3*5  The  Role  of  Source  Selection 

The  primary  roles  of  the  source  selection  process  are 
to  select  a contractor,  allocate  risk  between  the  government 
and  the  contractor,  and  to  provide  for  review  and  management 
by  the  government  once  a contract  is  signed  (Lenk,  1977). 

The  procurement  process  has  its  roots  in  the  purchase  of 
food,  soap  and  other  necessities  for  George  Washington’s 
Army  (Report  of  the  Commission  on  Government  Procurement, 
1972,  Appendix  G) . Various  contractual  mechanisms  have 
evolved  to  try  to  meet  most  effectively  the  government’s 
requirements  for  obtaining  specific  types  of  goods  and  ser- 
vices.  Waks  (I960)  traces  the  use  of  the  system  source 

2 

Appendix  G of  the  Report  of  the  Commission  on  Govern- 
ment Procurement  (1972)  provides  an  informative  history 
of  the  evolution  of  the  procurement  process. 

3 

Lenk  (1977)  provides  a good  overview  of  the  various 
contractual  mechanisms  used  by  the  government. 


selection  concept  to  the  spring  of  1955  when  it  was  first 
used  by  the  Air  Force.  He  explains  that  source  selection 
shifts  the  emphasis  from  making  a technical  analysis  of 
the  relative  merits  of  specific  design  proposals,  to  making 
an  overall  analysis  of  the  relative  ability  of  a select 

4 

group  of  sources  to  accept  a particular  design  assignment. 

The  literature  on  source  selection  suggests  that  there 
is  dissatisfaction  with  both  the  process  of  source  selection 
and  the  results  of  the  process  (Esposito,  1973?  P*  9;  Hynes, 
1977).  This  dissatisfaction  occurs  both  among  government 
and  industry  personnel  (Keeling,  1975,  pp.  129-130).  The 
significant  resources  required  have  not  paid  off  in  the 
achievement  of  the  outputs  desired  (Blue  Ribbon  Defense 
Panel,  1970;  Esposito,  1973,  p.  9). 

11.3.6  Issues  Related  to  Source  Selection 

In  reviewing  the  issues  raised  about  the  source  selec- 
tion process,  it  is  useful  to  divide  the  process  into  four 
stages.  A somewhat  simplistic  division  is  represented  in 
Figure  II. 2.  The  first  stage  incorporates  selecting  the 
type  of  contractual  mechanism  to  be  used,  writing  and  dis- 
tributing the  RFP  and  briefing  the  prospective  bidders. 

4 

The  term  System  Source  Selection  was  adopted  by  the 
Air  Force  for  a set  of  guidelines  for  accomplishing  this 
type  of  solicitation  and  selection. 
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The  second  stage  of  proposal  preparation  includes  the  deci- 
sions by  firms  on  whether  and  how  to  bid  and  the  actual 
writing  of  proposals.  The  third  stage  includes  the  recruit- 
ing and  briefing  of  the  proposal  evaluation  panel  and  con- 
ducting the  technical  evaluation  of  proposals  which  are 
received.  Summary,  review,  and  award  is  that  part  of  the  . 
process  which  includes  summarizing  the  results  of  the  panel 
meetings,  determining  the  competitive  range,  reviewing  cost 
proposals,  obtaining  clarifications,  negotiation,  signing 
the  contract,  and  debriefing  unsuccessful  bidders.  Though 
these  stages  are,  for  the  most  part  sequential,  they  do 
overlap  somewhat  in  time  (e.g.,  selection  of  the  panel  gen- 
erally occurs  before  proposal  writing  is  completed). 

It  is  generally  desirable  that  each  stage  be  conducted 
both  effectively  and  efficiently.  It  is  also  desirable 
that  each  stage  support  what  comes  later  (e.g.,  the  format 
required  for  proposals  may  affect  the  ability  of  the  panel 
to  accomplish  its  evaluation) . The  remainder  of  this  sub- 
section will  use  these  four  phases  to  highlight  what  the 
literature  raises  as  issues  relating  to  the  effectiveness 
and  efficiency  of  the  source  selection  process. 

11.3.6.1  Preparation  and  release  of  the  RFP  and  brief- 
ing potential  bidders.  The  RFP  has  been  described  as  being 
primarily  a business  action  (Esposito,  1973,  p.  9);  its 
purpose  being  to  provide  accurate  information  and  guidance 


in  a competitive  marketplace  (Esposito,  1973,  p.  11;  Morris, 
1973,  p.  19).  The  information  provided  and  the  form  for 
the  responses  should  allow  prospective  bidders  to  determine 
effectively  whether  to  bid  and  enable  them  to  convey  their 
capabilities  to  the  government  (Fox,  1974,  p 301;  Keeling, 
1975).  Hynes  (1977),  in  reporting  on  a survey  of  defense 
contractors,  indicates  that  there  is  a great  deal  of  dis- 
satisfaction with  the  RFP  as  a communication  instrument. 

One  of  the  first  tasks  when  preparing  an  RFP  is  to 
select  an  appropriate  contract  type.  Gordon  (1980)  argues 
strongly  for  realistically  matching  the  contract  terms  to 
the  expectations  for  the  system  and  the  inherent  technical 
and  financial  risks.  Systems  procurements  currently  tend 
to  make  use  of  cost  reimbursement  contracts  rather  than 
fixed  price  contracts  (Lenk,  1977).  It  is  intended  that 
this  mechanism  be  used  where  there  is  uncertainty  as  to 
what  the  final  product  is,  its  form,  how  much  it  will  cost 
and  the  quantity  to  be  produced  (Fox,  1974,  pp.  230-231; 
Skolnick,  1969).  Contractors  are  reimbursed  for  allowable 
direct  and  indirect  costs,  overhead  and  often  a fee  bas.ed 

c 

Anderson  (1969)  describes  how  problems  occurred  when 
Secretary  McNamara  required  the  Department  of  Defense  to 
procure  systems  using  fixed  price  contracts.  The  most  pub- 
licized case  is  the  use  of  Total  Package  Procurement  for 
the  C-5A  program. 
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upon  a percentage  of  the  negotiated  contract  price.  Systems 
procurements  also  tend  to  make  use  of  negotiated  rather 
than  advertised  procurements  (Fox,  197^,  p.  252;  Schnitzer, 
1973) • In  negotiated  procurements  both  price  and  other 
factors  (technical  management)  are  to  be  evaluated  in  deter- 
mining who  wins. 

A number  of  important  issues  relate  to  the  writing 
of  the  RFP.  One  of  the  first  to  arise  is  to  determine  who 
should  be  involved  in  writing  it.  Esposito  (1973,  p.  10) 
and  Skolnick  (1969,  p.  74)  argue  for  writing  of  the  RFP 
and  retention  of  the  source  selection  process  by  the  staff 
people  who  are  most  closely  concerned  with  the  specific 
aspects  of  the  program  and  its  management. 

The  basis  for  competition  among  bidders  must  be  deter- 
mined. Peck  and  Scherer  (1962,  pp.  343-355)  discuss  design 
competition  and  the  management  competition  as  two  ends  of 
a scale  for  the  technical  evaluation.  In  design  competition 
bidders  are  evaluated  on  the  basis  of  the  specific  designs 
which  they  propose  for  the  system.  In  management  competition 
the  evaluation  is  based  upon  the  approach  the  bidder  intends 
to  use  in  dealing  with  various  tasks  to  be  accomplished 
in  developing,  the  system  and  the  capability  of  the  proposed 


Lenk  (1977)  reviews  some  of  the  variations  on  these 
two  basic  types  of  contracts,  such  as  those  utilizing  var- 
ious incentive  schemes. 
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team  (Peck  & Scherer,  1962,  pp.  3^3,  354,  362,  367).  Helman 
and  Taylor  (1976,  p.  88)  indicate  that  often  bidder’s  manage- 
ment capability  is  not  adequately  covered  during  proposal 
evaluation . 

The  level  of  detail  of  advance  system  design  work  pro- 
vided in  the  RFP  must  also  be  determined.  If  the  government 
has  the  time  and  resources,  it  may  wish  to  develop  a detailed 
design  in  advance  of  issuing  an  RFP  and  require  bids  on 
developing  the  system  based  upon  this  design.  If  time  and 
resources  do  not  allow  this,  the  government  may  require 
bids  against  a set  of  performance  specifications  (Hall, 

1975,  p.  98).  Esposito  (1973,  p.  12)  indicates  that  a key 
problem  in  providing  information  on  requirements  is  to  re- 
flect and  deal  effectively  with  the  varying  levels  of  uncer- 
tainty. He  argues  for  separating  the  knowns  from  the  un- 
knowns by  doing  a good  job  of  defining  the  knowns  and  by 
identifying  the  unknowns  and  a management  approach  for  deal- 
ing with  them. 

A format  for  the  bidders’  responses  must  be  determined. 
Decisions  are  required  on  both  the  outline  and  the  allowable^ 
length  for  proposals.  Frequently  the  outline  is  arranged 
in  a manner  which  does  not  mesh  with  the  other  elements 
of  the  RFP  (i.e.,  the  statement  of  work,  the  evaluation 
criteria,  the  work  breakdown).  The  result  is  wasted  time 
and  poor  proposals  (Morris,  1973,  p.  23).  Morris  (1973, 
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p.  24)  advocates  using  an  outline  which  would  allow  propo- 
sals to  be  so  uniform  that  they  could  be  evaluated  almost 
on  a paragraph-by-paragraph  basis.  Many  people  also  advo- 
cate limiting  the  allowable  length  of  proposals.  Claims 
in  favor  of  this  are  that  proposals  can  be  evaluated  using 
fewer  government  resources  (Fox,  1974,  p.  266;  Skolnick, 

1969,  p.  82),  and  that  bidders  are  forced  to  concentrate 
on  the  most  important  issues  (Fox,  1974,  p.  266).  Logan 
(1962,  p.  32)  mentions  the  problem  of  brochuremenship  or 
"gold  plating"  of  proposals,  where  contractors  hope  to  make 
up  for  weakness  through  volume  and  fancy  art  work.  He  indi- 
cates that  this  tends  to  prejudice  the  evaluator  who  has 
to  "sift  through  the  chaff  in  order  to  find  the  wheat"  (Logan, 
1966,  p.  32).  An  alternate  view  of  this  problem  is  that 
bidders  resort  to  brochuremenship  or  "gold  plating"  because 
they  cannot  tell  from  the  RFP  what  information  the  govern- 
ment will  use  in  its  evaluation.  In  order  to  be  safe  they 
try  to  provide  information  on  every  issue  they  think  might 
arise  during  the  evaluation  panel’s  deliberations  (Thompson, 
1978a). 

Another  issue  which  is  closely  related  to  this  is  to 
decide  what  information  to  place  in  the  RFP  about  the  propo- 
sal evaluation  process.  An  argument  for  complete  informa- 
tion on  the  process  is  that  bidders  will  submit  more  respon- 
sive proposals  which  will  take  less  time  to  evaluate  (Hostler, 
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1973,  P 13;  Keeling,  1975,  p.  125;  Morris,  1973,  pp.  21- 
22).  A counter  argument  which  the  author  has  heard,  but 
has  not  noticed  in  the  literature,  is  that  telling  the  bid- 
ders the  criteria  for  evaluation  will  enable  them  to  tell 
you  only  what  they  think  you  want  to  hear,  thereby  not  giving 
you  a basis  for  discriminating  among  them. 

A final  issue  on  RFP  preparation  is  to  determine  how 
much  information  to  furnish  to  prospective  bidders.  Steward 
(1973)  argues  for  limiting  the  size  of  RFPs.  His  claim 
is  that  it  forces  better  writing,  creates  fewer  errors  of 
inconsistency,  can  be  read  and  understood  at  the  corporate 
officer  level,  and  that  it  limits  the  amount  of  information 
submitted  in  response  (Stewart,  1973,  p.  1^).  Another  ap- 
proach is  to  try  to  furnish  anything  which  may  help  to  mini- 
mize misunderstandings  (Thompson,  1978).  This  includes 
presentations  and  question  and  answer  sessions  at  bidders 
briefings . 

11.3.6.2  Proposal  preparation.  During  the  proposal 
preparation  phase  a prospective  bidder  must  determine  whether 
to  bid  and  if  they  decide  to  bid,  they  must  determine  what 
approach  to  take  in  writing  their  proposal  (Goldberg,  1977, 
p.  257).  Since  the  government  assumes  the  cost  of  evalua- 
tion of  proposals  and  most  of  the  cost  of  preparation  of 
proposals  (through  reimbursable  overhead  on  other  contracts), 
it  may  be  interested  in  a process  which  allows  firms  to 
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self  select  effectively  on  bidding  (Bailey,  1964,  p.  16; 

Roberts,  1964,  p.  76). 

Several  criteria  are  discussed  as  being  important  for 
a prospective  bidder  to  consider  in  determining  whether 
to  bid.  They  relate  to  market  strategy  and  whether  the 
firm  thinks  it  has  a good  chance  of  winning.  The  firm's 
decision  on  bidding  may  be  influenced  by  desire  to  develop 
or  maintain  a reputation  in  the  line  of  work  required  by 
the  RFP  and  by  factors  of  availability  of  company  resources 
(Peck  & Scherer,  1962).  If  bidding  is  consistent  with  the 
company's  market  strategy,  they  may  next  assess  their  chances 
of  winning  the  contract.  They  will  want  to  assess  their 
market  intelligence  to  determine  whether  the  contract  is 
"wired"  (i.e.,  whether  the  government  has  established  a 
sole  source  position  on  the  award)  (Fox,  1974,  p.  296), 
whether  they  have  a positive  image  with  the  program  person- 
nel (Fox,  1974,  p.  296;  Roberts,  1964,  p.  75),  and  whether 
the  actual  objectives  of  these  personnel  differ  from  what 
is  said  in  the  RFP  (Fox,  1974,  p.  296).  Bidders  may  also 
consider  the  resources  available  to  them  for  bidding  (i.e., 
time  of  specific  people  and  money)  and  whether  they  feel 
that  they  can  successfully  demonstrate  their  capability 
with  the  resources  that  they  are  able  to  apply  (Goldberg, 

1977,  p.  257). 


If  a firm  decides  to  bid,  it  may  decide  to  choose  one 
of  two  major  strategies.  One  is  to  try  to  submit  the  lowest 
bid  while  being  at  least  satisfactory  on  all  other  aspects 
of  the  proposal.  The  other  is  to  try  to  submit  the  superior 
proposal  and  to  have  a cost  in  the  competitive  range  (Fox, 
1974,  p.  296). 

11.3.6.3  Proposal  evaluation  panel  meets.  The  role 
of  the  proposal  evaluation  panel  is  to  assess  the  degree 
to  which  each  proposal  is  potentially  technically  acceptable 
on  each  of  the  evaluation  criteria.  Some  of  the  key  issues 
for  this  part  of  the  source  selection  process  are  recruiting 
and  preparing  the  panel,  dealing  with  the  large  amount  of 
information  which  must  be  processed  in  order  to  assess  suc- 
cess, determing  a scale  for  use  in  evaluating  proposals, 
deciding  what  information  is  admissible  for  evaluation  pur- 
poses, establishing  a process  for  obtaining  a group  evalua- 
tion, and  having  a method  for  dealing  with  missing  informa- 
tion. 

When  recruiting  an  evaluation  panel,  it  may  be  useful 
to  obtain  at  least  one  person  who  is  qualified  to  represent 
each  important  aspect  of  the  system.  Both  technical  experts 
and  program  participants  are  important  (Bureau  of  Naval 
Weapons,  1965,  pp.  8-9).'^  Carnes  ( 1976)  and  Logan  ( 1966) 

Roberts  (1964,  p.  74)  indicates  that  in  some  cases, 
the  technical  initiator  may  select  the  panel  with  the  ob- 
jective of  getting  participants  who  will  favor  his  preferred 
bidder . 


indicate  that  it  is  frequently  quite  difficult  to  obtain 
participation  of  experts  on  evaluation  panels.  The  time 
required  for  preparation  and  participation  may  prevent  ex- 
perts from  taking  part. 

Panel  members  are  often  responsible  for  sorting  through 
vast  amounts  of  information  which  is  present  in  proposals 
in  order  to  arrive  at  an  estimate  of  the  probability  that 
each  bidder  will  perform  effectively.  In  working  with  large 
amounts  of  information  people  are  constrained  by  limits 
on  their  ability  to  process  it  (Miller,  1956;  Weaver,  19^9). 
The  proposal  evaluation  process  will  generally  have  some 
method  for  chunking  information  and  then  integrating  scores. 

A form  of  rating  scale  must  also  be  determined.  Dycus 
(1977,  pp.  257-258)  identifies  five  general  categories  of 
rating  scale  as  numeric,  adjective-numeric  (e.g.,  1-10  scale 
with  qualitative  adjective  for  each  score),  check  off  (e.g., 
acceptable/unacceptable),  adjective  (e.g.,  color  codes), 
and  relative  rank  (i.e.,  ordering  from  best  to  poorest). 
After  comparing  them  for  various  scale  effects,  he  suggests 
that  "the  best  approach  might  be  a rating  scale  that  asks 
the  evaluators  to  consciously  make  both  absolute  and  rela- 
tive evaluative  judgments"  (Dycus,  1977,  p.  264). 

The  issue  of  admissible  evidence  is  primarily  one  of 
how  to  deal  with  bias  where  the  members  of  the  panel  know 


more  about  some  bidders  than  others.  Roberts  (1964)  points 
out  that  the  lack  of  bias  in  proposal  evaluation  is  impos- 
sible and  probably  undesirable.  He  indicates  that  technical 
"facts”  are  subject  to  dispute  by  experts  and  that  apprai- 
sals are  based  upon  judgments,  technical  prejudices,  and 
other  subjective  factors  (Roberts,  1964,  p.  74).  The  issue 
of  bias  is  how  to  recognize  it  and  incorporate  it  into  the 
evaluation  so  that  it  is  not  used  selectively  for  or  against 
only  certain  specific  bidders.  This  is  an  especially  diffi- 
cult problem  when  it  comes  to  incorporating  information 
about  a bidder’s  past  performance  into  the  evaluation. 

Nassr  (1978)  indicates  that  DOD  has  tended  to  place  insuffic 
ient  emphasis  on  doing  this.  Richardson  (1977)  quotes  an 
Army  General  as  saying  "the  concept  [use  of  past  performance 
in  evaluation]  is  unbelievably  difficult  to  implement  and 
there  does  not  appear  to  be  any  ready  formal  solution"  (Rich 
ardson,  1977,  p.  425).  Keeling  (1975,  p.  130)  reports  that 
the  methods  which  have  been  tried  for  doing  this  have  been 
based  on  historical  data  and  an  assumption  that  contractors 
are  monolithic.  He  argues  for  analyzing  past  performance 
based  upon  the  track  records  and  the  proposed  extent  of 
involvement  of  the  specific  personnel  on  the  company’s  pro- 
posed  project  team. 

The  evaluation  panel  meetings  are  a group  decision- 
making process  and  as  such  require  some  type  of  process 


for  determining  how  scores  are  assigned.  Voting,  reaching 
consensus,  and  leaving  the  decision  to  the  chairman  are 
three  possibilities.  Since  this  is  a group  decision-making 
process,  issues  of  how  the  ground  rules  affect  motivation 
and  effectiveness  are  relevant.  Some  important  issues  may 
include  whether: 

o the  group  members  feel  confident  that  the 
results  of  their  evaluation  will  be  used 
in  the  final  decisions  (Bradford,  Stock, 

& Horwi tz , 1953); 

0 there  is  a process  for  dealing  with  strong 
individual  disagreements  (Bradford,  Stock 
& Horwitz,  1953); 

o lower  status  individuals  who  feel  that  their 
opinions  are  better  than  higher  status  indi- 
viduals are  able  to  express  themselves  and 
prevail  (Torrance,  1957,  pp.  314-318); 

0 the  group  is  able  to  resist  undue  influence 
by  dominant  personality  types  (Dalkey  & Helmer, 
1963;  Roberts,  1964);  and 

o implied  threat  of  sanctions  from  more  knowl- 
edgeable members  is  controlled  (Dalkey  & 

Helmer,  1963 ) • 

A final  aspect  of  the  evaluation  panel  process  is  deter- 
mining how  to  deal  with  missing  information.  In  some  in- 
stances the  panel  will  not  understand  an  approach  being 
proposed  by  a bidder.  This  may  be  due  to  lack  of  informa- 
tion provided  in  the  proposal  (Carnes,  1976,  p.  7).  If 
the  panel  has  no  way  of  obtaining  this  missing  information, 
it  may  result  in  the  bidder  receiving  a lower  evaluation 
than  is  warranted  (Fox,  1974,  p.  273).  Carnes  (1976,  p. 
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7)  describes  how  incorrect  scoring  of  initial  proposals 
occurs  when  the  only  form  of  communication  is  through  a 
deficiency  reporting  process. 

11.3*6.4  Summary,  review  and  award.  After  the  panel 
has  completed  its  evaluation,  the  results  of  the  process 
are  summarized  in  written  form  and  submitted  to  the  contract- 
ing office  for  review,  clarification  and  negotiation.  The 
summary  is  generally  done-  by  the  chairman  of  the  evaluation 
panel  (often  the  Contracting  Officer’s  Technical  Representa- 
tive) who  also  recommends  those  bidders  who  may  be  techni- 
cally qualified  for  the  contract  after  clarification  and 
negotiation.  Often  the  program  will  have  a person  or  group 
of  people  above  the  level  of  the  panel  chairman  who  review 
his  report  and  recommendations  before  they  go  to  the  contract 
ing  office.  The  technical  evaluation  is  then  reviewed  to- 
gether with  an  analysis  of  the  cost  proposals.  Clarifica- 
tions are  obtained  from  bidders  if  necessary.  The  clarifica- 
tion process  may  result  in  some  bidders  being  excluded  from 
the  remaining  competition.  The  contracting  office  negoti- 
ates with  the  remaining  bidders  on  technical  considerations, 
price  considerations,  and  provisions  for  future  changes. 

The  contracting  officer,  after  consulting  with  technical 
people,  makes  a determination  of  the  offer  which  is  most 
favorable  and  signs  a contract.  The  signing  is  often  fol- 
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lowed  by  requests  for  debriefing  and  occasionally  protests 
of  the  award  by  unsuccessful  bidders. 

Determination  of  the  competitive  range  is  largely  with- 
in the  discretion  of  the  contracting  officer.  He  is  not 
bound  by  the  recommendations  of  the  evaluation  panel 
(Schnitzer,  1973).  The^  relevant  statute  (10  U.S.C.  2304(g)) 
requires  that  "an  offeror  is  held  to  be  in  the  competitive 
range  if  his  proposal  can  be  improved  reasonably  to  the 
point  where  it  becomes  the  most  acceptable."  In  practice 
this  is  often  a difficult  determination  to  make.  Scores 
of  competing  contractors  are  often  very  close.  The  large 
number  of  items  evaluated  often  levels  out  the  overall  rating 
(Blue  Ribbon  Defense  Panel,  1970).  In  other  cases  a bid- 
der’s overriding  superiority  in  a particular  area  may  satur- 
ate the  evaluation,  even  though  the  bidder  is  weak  in  other 
areas  of  the  proposal  (Skolnick,  1969,  p.  75).  The  numeri- 
cal scores  may  obscure  some  of  the  more  important  issues 
(Blue  Ribbon  Defense  Panel,  1970).  Schnitzer  (1973?  p. 

17)  stresses  that  competitive  range  should  be  decided  on 
the  basis  of  the  array  of  scores  received  rather  than  an 
overall  passing  grade. 

If  there  is  a higher  level  review  committee,  it  may 
disagree  with  the  recommendations  of  the  panel.  In  this 
case,  the  panel  may  be  asked  to  revise  their  report  until 
the  senior  group  agrees  with  it  (Carnes,  1976,  pp.  27-30; 


Roberts,  I960,  p.  75).  The  discussions  and  wisdom  of  the 
panel  are  ignored. 

Because  of  the  uncertainty  associated  with  information 
in  the  proposals,  clarifications  will  often  be  required 
before  a confident  technical  assessment  can  be  completed. 
Representatives  of  firms  in  the  competitive  range  are  called 
in  for  discussions  with  technical  and  contracting  office 
representatives.  Technical  leveling  is  a problem  which 
often  arises  here.  The  technical  people  may  wish  to  get 
all  firms  to  incorporate  approaches  which  they  found  favor 
with  in  the  proposals  of  other  firms.  When  firms  are  will- 
ing to  modify  their  proposals  to  comply,  there  is  little 
basis  for  discriminating  among  proposals  on  technical  grounds 
(Babione,  1978;  Carnes,  1976,  pp.  30-31;  Hall,  1975,  p. 

89;  Schnitzer,  1973,  p.  27). 

If  there  is  no  basis  for  technical  discrimination, 
the  result  is  a price  competition  (Carnes,  1976,  p.  2;  Fox, 
T974,  pp.  251,  277,  470;  Keeling,  1975,  p.  127;  Richardson, 
1977,  p.  423).  In  effect  this  turns  a negotiated  procure- 
ment into  an  advertised  procurement.  Award  is  generally 
made  to  the  low  bidder,  since  there  is  a requirement  to 
justify  the  choice  of  any  other  bidder  on  technical  grounds 
(Fox,  1974,  pp.  251,  277).  This  encourages  "buy-ins,"  pric- 
ing the  bid  low  in  order  to  get  the  contract  and  then  re- 
questing price  or  performance  modifications  when  it  is  clear 


that  the  promised  system  cannot  be  produced  at  that  price 
(Keeling,  1975,  p.  127;  Logan,  1966).  The  contractor  may 
use  various  techniques  to  come  up  with  a lower  cost,  includ- 
ing overstating  his  future  business  base  to  obtain  a lower 
overhead  rate  (Keeling,  1975,  p.  127)  and  using  less  costly 
personnel  than  required  to  do  the  work. 

Often  after  the  contract  is  signed  unsuccessful  bidders 
will  request  debriefings  to  discover  where  they  were  con- 
sidered weak.  Buying  agencies  sometimes  have  trouble  explain 
ing  the  basis  for  their  evaluation  (Peck  & Scherer,  1962, 
p.  372).  If  bidders  feel  that  they  were  evaluated  unfairly, 
they  may  file  a protest  of  the  award  with  the  General  Account 
ing  Office.  A successful  protest  may  result  if  the  govern- 
ment did  not  follow  its  stated  evaluation  procedure,  i.e., 
it  based  the  award  on  criteria  other  than  what  were  listed 
in  the  RFP  or  if  systematic  bias  entered  into  the  evaluation 
process.  Even  if  a formal  protest  is  not  filed,  a bidder 
who  feels  that  he  was  treated  unfairly  may  be  reluctant 
to  bid  on  future  work  by  the  government  agency. 

11.3*7  Summary  of  Systems  Procurement  Background 

This  section  has  reviewed  the  concept  of  system  and 
some  of  the  literature  related  to  the  acquisition  of  sys- 
tems. Emphasis  has  been  placed  on  describing  the  issues 
related  to  system  source  selection  which  are  raised  in  the 
literature.  The  next  section  describes  the  history  and 


characteristics  of  the  Parametric  Factor  Evaluation  source 


selection  process. 

g 

II . 4 Parametric  Factor  Evaluation 
II . 4 . 1 History 

The  Parametric  Factor  Evaluation  (PFE)  process  for 
source  selection  was  developed  in  the  mid~1950s  during  the 
early  days  of  the  weapons  system  management  concept  (Living- 
ston, 1959;  Thompson,  1976a,  pp.  169-170).  It  was  used 
by  the  Electronic  Reconnaissance  Section,  Aerial  Reconnais- 
sance Laboratory,  at  Wright  Field  in  the  procurement  of 
two  electronic  reconnaissance  subsystems  and  one  weather 
reconnaissance  subsystem  (Thompson,  1976a,  p.  170).  The 
process  was  developed  by  Charles  Thompson  to  meet  several 
specific  problems  in  trying  to  procure  these  subsystems. 

First,  the  requirements  (or  specifications)  were 
constantly  changing  because  of  the  need  to  respond 
to  new  developments  in  radar  (and  other  emitters). 
Second,  few  laboratory  personnel  had  any  signifi- 
cant prior  experience  in  designing  or  procuring 
systems.  And,  finally,  equipment  suppliers  had 
little  relevant  experience  (except,  in  some  cases, 
as  subcontractors  to  systems  contractors) (Thompson, 
1976a,  p.  170). 


g 

The  reader  who  is  interested  in  a more  detailed  de- 
scription of  the  Parametric  Factor  Evaluation  process  will 
wish  to  read  Thompson’s  (1976a)  description  which  is  repro- 
duced in  the  appendix.  The  article  was  drawn  upon  for  this 
section. 


Gon- 


The  existing  published  information  on  the  process  is 
tained  in  three  documents  (Thompson,  1976,  1976a;  U.  S. 

Air  Force , 1964)  . 

II. 4. 2 Claims 

Thompson  has  made  claims  for  the  circumstances  under 
which  PFE  will  and  will  not  be  useful  and  for  what  it  will 
accomplish.  He  has  indicated  that  the  process  is  applicabl 
to  large  and  complex  systems  (hardware  or  software)  where 
the  follov/ing  criteria  are  met: 

a)  that  there  is  a degree  of  uncertainty  or 
complexity  in  the  item  to  be  procured  which 
makes  it  difficult  to  describe  the  item 

to  bidders  and  to  identify  a suitable  basis 
for  evaluating  bidders’  proposals,  using 
available  procurement  methods; 

b)  that  there  are  limits  on  time,  money,  or 
personnel  which  preclude  considering  other 
alternatives;  and 

c)  that  there  is  some  basis  in  prior  experi- 
ence for  believing  that  it  is  critical 

to  both  the  procuring  agency  and  the  (suc- 
cessful) bidder  to  develop  a reasonably 
sound  initial  mutual  understanding  in  order 
to  ensure  a progressive  and  successful 
mutual  accomplishment  of  the  object  of 
the  procurement  (Thompson,  1976a,  p.  179). 

He  has  indicated  that  the  process  would  probably  be  inappli 

cable  to 


procurements  in  which  there  is  a single  (or 
small  number  of)  critical  basis(es)  for  evalua- 
tion, e.g.,  procurements  for  well-defined  pro- 
ducts or  services  where  the  basis  is  price, 
and  research  procurements  where  the  basis  is 
a particular  technical  approach  or  capability 
(p.  178). 


He  claims  that  in  general 


there  would  appear  to  be  no  need  for  this  pro- 
curement process  where  the  prior  experience 
of  the  procuring  agency  (and  the  set  of  pro- 
spective suppliers)  has  provided  a basis  for 
successful  procurement  and  especially  where 
there  are  comparable  previous  procurements 


(p. 

178). 

In  addition 

he  lists  other  procurement  situations  in  which 

alternat ive 

methods  are  being  used.  These  include: 

a) 

where  the  procuring  agency  has  the  time 
and  resources  to  carry  out  the  preliminary 
design  phase  (or  even  further  stages)  and 
prepare  detailed  specifications; 

b) 

where  the  procuring  agency  has  the  money 
(and  time)  to  let  parallel  (competitive) 
contracts  for  the  preliminary  design  (or 
concept)  phase; 

c ) 

where  the  procuring  agency  has  the  time 
(and  money)  to  let  a "sole  source"  contract 
for  the  preliminary  design  phase,  with 
the  flexibility  to  drop  the  program  or 
to  start  over  if  the  first  effort  was  unsuc- 
cessful ; 

d) 

where  one  or  more  prospective  suppliers 
are  willing  and  able  to  carry  out  the  pre- 
liminary design  phase  (or  even  further 
phases)  prior  to  action  by  by  procuring 
agency; 

e) 

where  the  item  to  be  procured  can  be  "decom- 
posed" from  technical  and  scheduling  inter- 
faces through  the  use  of  standards,  and 
the  like,  or  where  there  are  acceptable 
alternatives  available  (Thompson,  1976a, 
p.  178).  • 

II. 4. 3 Characteristics 

This  sub-section  describes,  in  summary  form,  the  activ 


ities  required  to  use  PFE  and  the  characteristics  which 


make  it  similar  or  dissimilar  to  other  source  selection 


processes . 

The  activities  required  to  use  PFE  are  listed  below. 
They  are  approximately  in  the  order  in  which  they  occur 
in  time. 

1)  Assess  whether  PFE  is  appropriate  for  the 

' proposed  procurement.  Is  it  appropriate 

in  terms  of  the  characteristics  of  the 
thing  to  be  procured,  the  procuring  agency, 
and  the  set  of  potential  bidders? 

2)  Review  what  is  known  about  the  procurement 
to  identify  the  conditions  which  help  to 
define  what  it  is  you  are  going  to  procure. 

3)  Refine  this  information  into  a series  of 
statements  describing  the  characteristics 
of  a prospective  bidder/contractor  which 
you  believe  critical  to  the  program’s  suc- 
cess. This  should  result  in  20-70  indi- 
vidual statements,  called  sub-factors, 
written  to  minimize  gaps  and  overlaps. 
Additional' specific  illustrative  questions 
can  be  added  for  clarity. 

4)  Group  the  sub-factors  into  5-15  major  areas 
of  concern  called  factors.  These  are  ad- 
ministrative groupings  selected  to  reflect 
related  problems  and  related  interests 

and  skillsof  people  who  will  serve  on 
the  evaluation  panel. 

5)  Prepare  the  "Statement  of  Work"  and 
"Schedule"  specifying  the  output  of  the 
project  according  to  the  format  provided 
by  the  factor  structure. 

6)  Prepare  the  "Scope  of  Effort"  document 
following  the  same  format  to  provide  a 
framework  for  the  bidder  to  present  his 
estimate  of  how  he  will  allocate  manpower 
and  dollars  to  various  parts  of  the  program. 
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7)  Gather  copies  of  supporting  documents  to 
provide  bidders  with  any  information  which 
might  be  helpful  in  preparing  proposals. 

This  step  can  occur  at  other  stages,  as 
long  as  this  material  is  available  to  bid- 
ders when  the  RFP  is  released. 

8)  Prepare  the  "Proposal  Preparation  Proce- 
dure" to  explain  the  reason  for  using  the 
PFE  process,  the  format  for  response,  and 
how  the  proposals  will  be  evaluated. 

9)  Combine  all  of  the  documents  with  the  formal 
parts  of  the  RFP  to  assemble  the  entire 
package . 

10)  Brief  the  bidders  by  making  presentations 
on  the  purposes  of  the  project  and  use 

of  the  PFE  technique.  Encourage  questions 
from  those  present.  If  PFE  is  a departure 
from  past  practice,  it  is  desirable  to 
hold  two  briefings.  One  to  allow  bidders 
to  decide  whether  or  not  to  bid.  The  second 
primarily  to  answer  questions. 

11)  Prepare  the  "Proposal  Evaluation  Procedure" 
to  provide  detailed  guidelines  for  members 
of  the  evaluation  panel. 

12)  Select  members  of  the  evaluation  panel 
from  those  concerned  directly  with  the 
success  of  the  program  and  from  individuals 
whose  background  and  skills  are  necessary 
for  the  evaluation  of  particular  factors. 

It  is  desirable  for  as  many  panel  members 
as  possible  to  participate  in  developing 
the  list  of  sub-factors.  Early  selection 
is  recommended. 

13)  Panel  members  prepare  for  the  evaluation 

by  familiarizing  themselves  with  the  entire 
RFP  including  the  supporting  documentation. 
Panel  members  are  invited  to  participate 
in  briefing  the  prospective  bidders. 

14)  Proposals  are  received  and  reviewed  by 
panel  members  for  preparation  of  initial 
ratings  of  bidders  on  the  specific  sub- 
factors to  which  they  are  assigned. 


15)  Bidders  make  a presentation  on  their  pro- 
posals to  the  panel.  Panel  members  are 
encouraged  to  question  bidders. 9 

16)  Factor  teams  meet  to  arrive  at  a single 
aggregate  rating  of  each  bidder.  Teams 
arrive  at  ratings  for  each  sub-factor  based 
upon  criteria  of  bidder’s  comprehension 

or  understanding  of  the  problem  presented, 
approach  described,  and  present  or  poten- 
tial capability  to  perform.  Panel  members 
turn  in  signed  rating  forms  before  the 
discussion  begins.  The  panel  arrives  at 
agreement  in  any  manner  it  chooses,  exclud- 
ing voting.  The  panel  rating  forms  are 
signed  by  all  team  members,  with  strong 
individual  disagreements  and  specific  com- 
ments for  the  negotiation  process  recorded 
on  the  form.  All  meetings  are  tape  recorded. 

17)  Prepare  a summary  of  all  factor  ratings 
for  use  in  final  review  and  negotiation. 

Thompson  (1976a)  indicates  the  set  of  general  and  spec- 
ific characteristics  which  when  combined  make  the  PFE  pro- 
cess unique.  The  overall  characteristics  are: 

1)  The  process  is  not  applicable  everywhere. 

It  is  designed  to  meet  a particular  kind 
of  procurement  problem. 

2)  The  process  is  pragmatic  in  proposing  that 
uncertainty  be  faced  and  not  avoided. 

Bidders  are  told  the  degree  of  uncertainty 
in  the  program  personnel’s  knowledge. 

3)  The  process  is  open,  explicit,  and  coopera- 
tive in  exchanging  information  and  preserv- 
ing records. 
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ETIP  has  substituted  a written  question  and  response 
mechanism  for  the  oral  presentation.  Panel  members  who 
require  clarifications  prior  to  their  individual  ratings 
submit  questions  to  the  bidder  through  the  contracting  of- 
fice. 


The  specific  characteristics  are: 

1 ) The  process  neither  assumes  nor  requires 
any  formal  structure  for  the  object  of 
the  procurement.  The  sub-factors  are 
treated  as  separate  "probes,”  obtaining 
measures  along  a set  of  dimensions  which 
describe  the  program. 

2)  The  process  uses  a single  framework  for 
all  of  its  documents,  i.e.,  the  factor 
structure . 

3)  No  limitation  is  placed  on  length. 

4)  The  evaluation  team  is  formed  from  among 
those  who  have  a prior  interest  in  the 
subject  of  the  procurement  or  who  have 
special  competence  to  evaluate  particular 
sub-factors.  Participation  may  be  limited 
to  individual  sub-factors. 

5)  There  are  not  restrictions  on  information 
which  the  evaluation  panel  may  use. 

6)  Members  of  the  evaluation  team  must  rate 
each  bidder  affirmatively,  i.e.,  have  a 
basis  for  both  low  and  high  ratings.  It 
cannot  assign  a "zero"  but  must  obtain 
required  information  through  a specific 
question. 

7)  The  process  of  aggregating  ratings  is  se- 
quential with  a summary  hard-copy  record 
retained  at  each  stage.  Ratings  are  aggre- 
gated from  individual  team  member  ratings 

of  sub-factor  to  team  ratings  of  sub-factors 
and  from  team  ratings  of  sub-factors  to 
team  ratings  of  factors.  The  panel  may 
use  any  process  for  doing  this,  excluding 
voting.  Factor  scores  are  then  aggregated 
to  an  overall  score  using  weighted  sums. 

The  sub-factor  rating  approach  is  contrasted  with  other 
approaches.  One  approach  is  that  which  is  used  in  the  case 
where  there  may  be  a comparably  high  degree  of  uncertainty 
on  the  part  of  the  procuring  agency,  but  it  may  be  able 


to  limit  the  evaluation  to  a "single”  or  small  set  of  techni 
cal  problems,  together  with  an  evaluation  of  the  capability 
of  the  individuals  who  work  on  the  project.  This  may  often 
be  the  case  in  the  procurement  of  research  (Thompson,  1976a) 
The  other  extreme  is  where  uncertainty  is  low  and  the  evalua 
tion  may  be  based  upon  a single  question.  An  example  of 
this  situation  is  production  procurements  to  detailed  speci- 
fications which  are  evaluated  on  the  question  , of  price, 
together  with  proof  of  the  existence  of  the  required  produc- 
tion facilities  (Thompson,  1976a). 

II . 4 . 4 ' Summary 

This  section  has  reviewed  the  PFE  process  in  terms 
of  its  history,  claims  for  where  it  is  and  is  not  useful, 
and  characteristics  for  using  it.  This  is  intended  as  a 
brief  summary  of  more  comprehensive  descriptions  which  are 
reproduced  in  the  appendix. 

II . 5 Summary 

The  purpose  of  this  chapter  was  to  provide  background 
on  the  general  problem  of  procurement  of  evaluation  systems 
and  on  the  specific  method  which  ETIP  chose  to  procure  its 
evaluation  systems.  The  intention  was  to  give  the  reader 
a basis  for  understanding  the  utility  of  this  research  in 
terms  of  problems  that  exist  and  in  terms  other  research 
which  has  come  before.  It  was  also  intended  to  provide 
sufficient  information  on  the  Parametric  Factor  Evaluation 


process  to  enable  the  reader 
presented  in  the  case  study. 
As  review,  the  starting 

that : 


to  understand  the  information 


point  for  this  research  is 


0 


the  evaluation  community  is 
recognize  the  importance  of 
tion  process; 


beginning  to 
the  source  selec- 


o the  systems  procurement  community  has  done 
much  writing  on  problems  with  the  source 
selection  process  and  of  guidelines  for  sol- 
ving it.  A number  of  case  studies  exist 
but  they  are  of  a summary  nature. 


o The  writing  on  PFE  is  limited  to  articles 
by  Thompson.  No  evaluations  of  the  use  of 
the  process  exist. 


The  next  chapter  describes  the  objectives  of  the  author 
in  furthering  this  existing  state  of  knowledge  and  provides 
a description  of  the  methods  used. 
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CHAPTER  III 

OBJECTIVES  OF  THE  RESEARCH 


111.1  Introduction 

This  chapter  is  intended  to  be  a guide  to  the  central 
purposes  of  the  research.  It  describes  what  the  author 
set  out  to  accomplish  and  the  approach  taken  in  meeting 
these  objectives.  A discussion  of  the  proposed  value  of 
this  research  and  a disclosure  of  the  biases  of  the  author 
which  might  affect  the  outcome  are  also  given. 

111. 2 Purpose  of  this  Research 

The  primary  purpose  of  this  research  is  to  provide 
a detailed  case  study  of  an  experiment  in  the  selection 
of  contractors  to  design  and  develop  systems.  Specifically, 
it  is  an  evaluation  of  the  use  of  the  Parametric  Factor 
Evaluation  approach  by  the  Experimental  Technology  Incen- 
tives Program  to  select  contractors  for  the  acquisition 
of  two  evaluation  systems.  These  source  selections  occurred 
during  the  1976  fiscal  year. 

The  case  study  is  based  upon; 

o background  on  the  PFE  process  from  both  historical 
documents  and  personal  communications  with  its  pri- 
mary developer; 


o 


participation  in  the  process  at  ETIP  by  the  author 
and  direct  access  to  records  of  the  source  selection 
of  the  two  contractors;  and 

o interview  and  quest ionnaire  information  from  other 
participants  in  the  source  selection. 

The  author’s  intention  is  to  provide  a full  description, 

with  data  from  the  process,  of  ETIP’s  use  of  the  PFE  process 

and  to  evaluate  how  successful  the  process  was  in  solving 

some  of  the  important  problems  which  existed.  It  is  hoped 

that  this  will  help  those  with  similar  problems  to  assess 

whether  the  technique  is  applicable  to  their  own  situation. 

The  selective  sampling  of  the  literature  on  source 
selection  and  procurement,  partially  reviewed  in  the  last 
chapter,  revealed  an  absence  of  detailed  case  studies. 

The  closest  similar  case  material  found  is  in  the  form  of 
summary  recollections  of  the  author  without  specific  data 
from  the  process  (Carnes,  1976).  Most  of  the  literature 
appears  to  be  in  the  form  of  prescriptive  guidelines  follow- 
ing discussions  of  problems  with  the  source  selection  pro- 
cess or  problems  in  the  work  of  the  resulting  contractors. 
That  literature  which  is  centered  on  ”how  to”  often  provides 
only  the  most  general  guidance  for  the  potential  user. 

There  is  little  offered  in  the  form  of  detailed  procedures 
to  relate  these  guidelines  to  the  complexity  of  an  actual 
procurement . 

This  research  is,  in  part,  an  attempt  to  begin  to  fill 


in  some  of  the  gaps  in  the  source  selection  literature. 


It  is  intended  to  be  a relatively  full,  self-conscious  recon 
struction  of  the  PFE  process  as  it  was  used  by  ETIP,  pre- 
sented together  with  the  actual  data  from  the  evaluation. 
This  is  follov/ed  by  an  evaluation  of  how  well  the  process 
controlled  for  some  of  the  key  problems  which  ETIP  faced 
in  the  procurement. 

III. 3 Format  Followed  in  the  Case  Study 

The  case  study  is  accomplished  largely  in  the  next 
two  chapters,  with  the  final  chapter  used  to  discuss  the 
relationship  of  the  process  to  issues  of  broader  signifi- 
cance than  the  ETIP  procurements. 

Chapter  IV  is  a relatively  full  description  of  ETIP’s 
application  of  the  PFE  technique  to  the  selection  of  con- 
tractors for  the  acquisition  of  the  two  evaluation  systems. 
Information  is  presented  on  background  leading  up  to  the 
writing  of  the  RFP,  the  roles,  extent,  and  timing  of  involve 
ment  of  various  people  in  the  process,  and  issues  which 
arose  at  various  stages  of  the  process.  Both  anecdotes 
and  samples  of  data  from  the  proposal  evaluation  process 
are  given  in  the  text.  All  of  the  initial  and  final  ratings 
from  the  evaluation  are  provided  in  the  appendix. 

The  process  followed  in  writing  Chapter  IV  was  to  draw 
upon  various  sources  of  information,  including: 
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o the  author’s  recollections; 

o the  formal  records  of  the  process  (including 
tapes  of  the  evaluation  panel  meetings  and 
transcripts  of  the  bidders’  briefings); 

o notes  from  the  files  of  the  Contracting  Offi- 
cer’s Technical  Representative  (COTR)  and  the 
author;  and 

o interviews  with  the  COTR  and  with  the  author 
of  the  RFP; 

to  write  a draft  chronology  of  the  process.  This  chronology 
was  then  given  to  the  COTR  and  to  the  author  of  the  RFP 
to  validate  the  statements  made,  to  obtain  additional  cross 
.validation  of  events  which  occurred  and  issues  which  arose, 

< and  to  obtain  additional  information  to  fill  in  holes  in 
the  chronology.  Later  interviews  with  other  participants 
were  also  used  to  obtain  additional  information  on  undocu- 
mented events.  The  emphasis  in  the  chronology  is  to  describe 
the  process  in  a manner  which  would  readily  answer  some 
of  the  basic  questions  which  a practitioner  might  have. 

These  would  include: 

o What  was  ETIP  trying  to  do? 
o What  did  ETIP  actually  do? 

Chapter  V is  outcome-oriented.  It  presents  an  evalua- 
tion of  how  well  the  PFE  process  worked  in  solving  many 
of  the  critical  issues  faced  by  ETIP  which  were  raised  by 
participants  or  are  raised  by  the  literature  on  source  selec- 
tion. Actual  data  from  the  process  is  used  in  assessing 
PFE’s  performance  against  each  of  the  issues  evaluated. 


Where  external  standards  exist  in  the  literature  or  were 
raised  by  participants,  they  are  compared  to  ETIP’s  exper- 
ience. For  issues  where  the  author  did  not  consider  himself 
an  acceptable  source  of  information,  either  through  lack 
of  access  to  the  required  information  or  through  his  bias, 
additional  information  was  solicited  from  other  participants 
through  questionnaires  and  interviews. 

The  chapter  presents  a list  of  issues  which  were  con- 
sidered for  the  evaluation.  A sub-set  of  these  issues  was 
selected  for  thorough  consideration.  The  choice  was  based 
on  judgments  of  both  the  importance  of  the  issue  and  the 
availability  of  information  for  evaluation  of  the  issue. 
Next,  the  set  of  parameters,  the  factors  which  may  affect 
the  interpretation  of  the  results,  is  presented.  The  po- 
tential impact  of  each  one  is  described.  Each  selected 
issue  is  then  evaluated.  The  issue  is  described,  a conclu- 
sion about  ETIP’s  experience  in  using  PFE  to  deal  with  the 
issue  is  reached,  and  the  data  supporting  this  conclusion 
are  presented.  After  all  of  the  issues  are  evaluated  indi- 
vidually, the  results  of  using  the  process  are  summarized 
in  terms  of  successes,  failures,  and  issues  where  the  re- 
sults are  uncertain. 

The  issues  in  Chapter  V against  which  PFE  is  evaluated 
are  those  which  relate  solely  to  the  source- selection  part 
of  the  system  acquisition  process  and  not  to  the  outcome 


of  the  projects  for  which  contractors  were  selected.  An 
evaluation  of  how  accurately  PFE  predicted  the  performance 
of  the  successful  bidders  would  have  been  a useful  addition 
to  this  research.  Unfortunately,  time  series  evaluations, 
by  the  government,  of  the  contractor’s  performance,  were 
not  collected  and  the  author  decided  not  to  attempt  to  do 
a post  hoc  reconstruction  of  this.  However,  there  are  some 
comments  in  Chapter  VI  related  to  subsequent  performance. 

Chapter  VI  is  a presentation  of  implications  of  ETIP’s 
use  of  PFE  and  the  results  of  this  case  study  in  terms  of 
subsequent  events,  potential  further  applications  of  PFE, 
and  potential  future  research.  Subsequent  events  include 
a brief  review  of  results  of  the  two  evaluation  system  acqu 
sitions  and  a brief  description  of  ETIP’s  later  uses  of 
PFE  for  other  source  selections.  Potential  further  applica 
tions  of  PFE  for  both  source  selection  and  other  problems 
are  speculated  upon.  Finally,  some  suggestions  are  made 
for  possible  related  research  on  topics  beyond  the  scope 
of  this  case  study. 

III. 4 Value  of  this  Research 

This  project  was  undertaken  at  the  request  of  the  for- 
mer Director  of  ETIP.  After  the  two  contracts  for  the  eval 
nation  systems  had  been  signed,  he  asked  the  COTR  to  eval- 
uate how  well  the  process  worked.  At  that  time  the  author, 
an  employee  of  the  COTR,  was  trying  to  find  a suitable  dis- 
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sertation  topic.  An  evaluation  of  the  PFE  technique  seemed 
as  though  it  might  meet  the  requirements  for  a disserta- 
tion and  at  the  same  time  satisfy  ETIP's  desire  to  evaluate 
how  well  the  process  worked.  ETIP  has  recently  been  going 
through  a reorientation  and  the  acquisition  of  evaluation 
systems  will  probably  not  be  an  objective  of  the  new  organ- 
ization. It  is  therefore  useful  to  explore  the  value  of 
this  project  for  both  the  researcher  and  for  those  inter- 
ested in  acquiring  evaluation  systems.  A claim  and  counter- 
claim will  be  used  to  try  to  illustrate  the  value  for  both 
audiences . 

III. 4.1  Research  Value 

The  claim  for  researchers  was  made  by  Donald  Campbell 
and  Julian  Stanley  in  their  classic  survey  of  designs  for 
research  (Campbell  & Stanley,  1966).  In  their  discussion 
of  the  one-shot  case  study,  the  method  used  in  this  research, 
they  state  that: 

such  studies  have  such  a total  absence  of  control  as 
to  be  of  almost  no  scientific  value  (Campbell  & Stanley, 

1966 , p . 6 ) . 

They  further  state  that: 

It  seems  well-nigh  unethical  at  the  present  time  to 
allow,  as  theses  or  dissertations  in  education, 
case  studies  of  this  nature  (i.e.,  involving  a 
single  group  observed  at  one  time  only )( Campbell 
& Stanley,  1966,  p.  7). 


Finally  they  propose  that: 


The  many  uncontrolled  sources  of  difference  be- 
tween a present  case  study  and  potential  future 
ones  which  might  be  compared  with  it  are  so  numer- 
ous as  to  make  justification  in  terms  of  providing 
a bench  mark  for  future  studies  also  hopeless 
(Campbell  & Stanley,  1966,  p.  7). 

The  counter-claim  for  this  research  is  that  it  will  lead 
to  judgments  about  the  effectiveness  of  the  technique  being 
tested  and  that  it  will  serve  as  a bench  mark  for  those 
interested  in  improving  upon  and  evaluating  PFE.  It  will 
also  set  a bench  mark  for  those  interested  in  evaluation 
of  source  selection  techniques. 

This  research  is  the  first  evaluation  of  an  application 
of  the  PFE  technique.  Though  PFE  was  developed  and  used 
several  times  by  the  Air  Force  in  the  1950’s,  there  is  no 
published  evaluation  of  its  effectiveness.  Also  this  was 
the  first  application  of  the  technique  in  a very  different 
setting.  In  addition,  to  the  author’s  knowledge,  no  care- 
ful, well  defined,  evaluation  of  any  system  source  selection 
technique  exists.  The  existing  literature  on  source  selec- 
tion describes  many  of  the  issues  which  are  important;  it 
examines  problems  which  have  occurred  in  past  procurement. 
The  literature  serves  as  a departure  point  to  enable  the 
author  to  specify  these  issues  and  use  them  to  structure 
an  evaluation  for  system  source  selection  techniques.  This 
case  study  evaluation  sets  up  a standard  for  comparative 
evaluations  of  other  techniques.  .Both  the  issues  selected 
and  the  method  for  evaluation  may  be  improved  upon. 


When  little  or  no  information  exists  on  a topic  as 
broad  as  the  one  considered  here,  the  traditional  ’’controls" 
applied  to  a priori  proposition  testing  are  inapplicable. 

A good  description  and  understanding  of  how  the  process 
works  are  important  first  steps  for  later  more  controlled 
research.  The  investigative  method  used  in  this  research 
is  felt  to  provide^  a scientific  basis  and  point  of  compari- 
son for  future  research  on  source  selection  procedures. 

III. 4. 2 Administrative  Value 

The  claim  for  those  interested  in  acquiring  evaluation 
systems  was  made  by  one  of  the  bidders  from  ETIP’s  source 
selection  during  an  interview  for  this  evaluation.  His 
claim  was  that,  as  a taxpayer,  this  research  is  "far  too 
expensive"  and  the  results  "too  insignificant"  to  justify 
it.  The  reasons  he  gave  for  his  claim  were  related  to  his 
beliefs  about  how  "policy  change"  gets  made  in  Washington. 

He  indicated  that  he  felt  that  in  an  emergency,  such  as 
a war,  people  would  move  things  rapidly  in  the  direction 
of  improving  source  selection.  He  indicated  that  whatever 
the  results  of  this  study,  a thoughtful,  reflective  commit- 
tee behind  a powerful  mover  could  have  implemented  it  at 
considerably  less  cost  and  in  greatly  reduced  time. 

The  counter-claim  comes  from  the  apparent  differences 
between  the  author’s  beliefs  and  the  bidder’s  beliefs  about 
how  change  is  caused  in  government.  The  author  believes 


that  high  level  directives  alone  will  not  lead  to  meaningful 
change.  It  is  primarily  the  bureaucrats,  those  people  who 
survive  changes  in  administrations  and  who  are  responsible 
for  translating  vague  policy  directives  into  specific  proce- 
dures, who  are  the  real  movers  in  government.  The  author 
has  experienced  and  observed  how  directives  from  the  top 
get  permanently  delayed  in  red  tape  because  those  respon- 
sible for  implementing  the  directives  oppose  them.  This 
is  not  to  say  that  support  from  the  top  is  unimportant; 
however,  the  author  believes  that  the  key  to  change,  at 
least  in  civilian  agencies,  is  the  user,  the  person  who 
has  to  live  with  the  change.  Change  occurs  when  a person 
has  a problem,  pressure  to  solve  it,  and  a solution  in  which 
he  has  confidence. 

The  literatures  on  evaluation,  systems  and  source  selec 
tion  clearly  indicate  that  there  are  problems.  These  prob- 
lems become  specific  to  users,  i.e.,  government  project 
managers,  contracting  officers,  and  other  government  techni- 
cal personnel  when  they  are  trying  to  acquire  specific  sys- 
tems . 

The  pressure  to  solve  these  problems  is  not  quite  as 
obvious,  particularly  where  evaluations  are  concerned. 

There  is  unlikely  to  be  an  emergency  which  would  impact 
this  type  of  work.  The  most  direct  pressure  seems  to  have 
come  from  Congress,  the  Office  of  Management  and  Budget, 


and  the  intended  users  of  the  systems.  In  some  cases  it 
comes  from  a high  level  administrator  and  in  other  instances 
from  the  independent  desire  of  the  users  of  source  selection 
methods  to  do  better. 

The  intent  and  thus  value  of  this  research  is  to  in- 
crease awareness  and,  through  a credible  evaluation,  modify 
confidence  in  a potential  solution  to  source  selection  prob- 
lems. Increased  awareness  is  provided  by  the  description 
of  the  process,  its  history,  and  the  types  of  problems  for 
which  it  might  be  applicable  (Section  II. 4),  and  the  full 
description  of  how  it  was  implemented  at  ETIP  (Chapter  IV). 
The  evaluation  of  how  well  ETIP  was  able  to  deal  with  the 
key  issues  of  source  selection  by  using  the  PEE  process 
(Chapter  V)  is  intended  to  enable  a user  to  judge  how  confi- 
dent he  is  that  the  process  will  be  useful  for  solving  his 
own  source  selection  problems. 

The  author  agrees  with  the  bidder  that  this  research 
alone  will  probably  not  convince  anyone.  The  type  of  confi- 
dence required  to  use  a technique  such  as  this  comes  f<rom 
either  or  both  trust  in  the  people  advocating  the  approach 
(ETIP’s  reason)  or  the  ability  to  verify  independently  for 
oneself  that  it  solves  the  problems  one  has.  ■ The  goal  here 
is  to  provide  the  ability  to  verify  whether  the  process 
works.  Clearly,  it  is  not  a totally  independent  verifica- 
tion since  the  reader  is  dependent  on  the  author’s  recon- 
struction and  interpretation.  However,  a great  deal  of 


raw  data  from  the  process  is  provided  to  enable  the  reader 
to  reach  his  own  conclusions  on  where  the  process  did  and 
did  not  work.  This  research  is  offered  as  a departure  point 
for  those  who  may  consider  using  or  improving  upon  the  PFE 
process . 

III. 5 Biases  of  the  Author 

When  examining  the  case  study,  the  reader  may  find 
it  helpful  to  keep  the  biases  of  the  author  in  mind.  These 
certainly  have  had  some  effect  on  the  presentation  of  the 
problems,  the  emphasds  in  the  description  of  events,  and 
the  evaluation  of  results.  This  section  is  presented  as 
an  attempt  to  disclose  the  author's  self-conscious  biases. 

The  author  is  not  an  expert  in  procurement,  though 
he  has  done  some  examination  of  the  literature  and  talked 
with  those  involved  in  the  process  (i.e.,  contracting  of- 
ficers, project  managers,  and  contractors).  Procurement 
is  not  an  area  in  which  he  claims  special  expertise.  He 
has  no  background  in  contract  law,  and  only  a modest  back- 
ground in  procurement  policies  and  the  practice  of  procure- 
ment . 

The  author  does  claim  special  knowledge  in  what  occur- 
red in  the  specific  procurements  under  examination  here. 
Working  on  these  procurements  was  the  author's  first  assign- 
ment when  he  arrived  at  ETIP  in  March  1976.  He  was  involved 


in  recruiting  members  for  the  evaluation  panel,  scheduling 
of  meetings,  evaluating  proposals,  and  assisting  in  the 
oral  and  written  clarification  process.  He  has  participated 
in  the  use  of  the  PFE  process  twice  subsequently  and  had 
an  active  role  in  writing  portions  of  the  RFPs.  He  does 
not  have  personal  experience  with  any  other  procurement 
techniques,  though  he  has  discussed  with  others  their  exper- 
iences. 

Finally,  and  possibly  most  significantly,  the  PFE  pro- 
cess was  primarily  developed  by  the  author's  dissertation 
advisor.  Aside  from  the  obvious  potential  bias  caused  by 
wanting  to  pass,  there  is  the  additional  factor  that  the 
author’s  advisor  has  greatly  shaped  his  values  about  what 
the  important  problems  of  contracting  are  and  how  to  solve 
these  problems.  This  has  been  a non-trivial  influence. 

This  disclosure  is  intended  as  a caveat  to  the  reader. 
The  author  has  attempted  to  compensate  for  these  biases 
by  several  methods.  Raw  data  from  the  evaluation  process 
are  provided  in  both  Chapter  IV  and  in  the  Appendix.  The 
reader  is  encouraged  to  examine  it  and  to  do  his  own  indepen 
dent  analyses.  Extensive  use  is  made  of  reference  notes 
in  Chapter  IV  to  indicate  sources  other  than  the  author’s 
own  notes  and  recollections.  Where  available,  cross  ref- 
erences are  used.  Additional  data  collected  from  panel 
members  and  bidders  is  used  in  Chapter  V in  the  evaluation 
of  some  of  the. issues.  Where  the  author  felt  that  he  might 
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not  be  a credible  source  of  evaluation  information,  he  sought 
additional  data  through  questionnaires  given  to  other  par- 
ticipants in  the  process. 

III. 6 Summary 

This  chapter  has  presented  a guide  to  the  objectives 
of  this  dissertation.  It  has  described  what  the  objectives 
are  and  the  process  being  followed  to  achieve  them.  The 
claimed  value  of  this  research  has  been  presented  and  the 
biases  of  the  author,  which  are  parameters  to  interpretation 
of  results,  have  been  disclosed. 

Chapter  IV  is  intended  to  describe  how  ETIP  used  PFE 
to  try  to  solye  its  procurement  problem.  It  presents  much 
of  the  factual  context  for  the  eyaluation  portion  of  the 
case  study  which  follows  in  Chapter  V.  If  the  reader  has 
not  done  so  already,  it  might  be  useful  before  proceeding, 
to  glance  at  the  sections  of  the  appendix  which  contains 
copies  of  the  RFP  and  of  Thompson’s  (1976a)  paper  describing 
the  PFE  process.  This  will  provide  a useful  context  for 
the  Chronology. 
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CHAPTER  IV 
CHRONOLOGY 


IV, 1 Introduction 


This  chapter  describes  how  Parametric  Factor  Evaluation 
(PFE)  was  actually  used  at  ETIP  in  1975  - 1976  to  procure 
the  two  evaluation  systems  contracts.  The  description  is 
based  on: 

o background  on  the  PFE  process  available  in 
the  form  of  historical  documents  and  descrip- 
tions by  the  primary  developer; 

o the  author’s  observations  during  his  partici- 
pation in  ETIP’s  use  of  PFE  and  access  to 
both  the  formal  records  of  the  process  and 
the  informal  notes  of  some  key  ETIP  partici- 
pants; and 

0 interviews  with  some  key  participants,  includ- 
ing bidders  and  members  of  the  evaluation 
panel . 

The  author’s  involvement  in  the  procurement  began  two 
weeks  prior  to  the  second  bidder’s  briefing  (March  8,  1976), 
during  the  selection  of  the  proposal  evaluation  panel. 
Sources  for  information  on  events  prior  to  this  date  are 
the  records  and  recollections  of  others.  The  author  did 
not  decide  to  use  this  problem  as  the  topic  for  a disserta- 
tion until  about  two  months  after  the  contracts  were  signed. 
Therefore,  no  special  effort  had  been  made  to  set  up  a con- 
temporaneous data  collection  scheme.  Fortunately,  the  Con- 


tracting  Officer’s  Technical  Representative  (COTR)  kept 
extensive  files  including  his  notes  of  meetings  and  phone 
conversations,  and  copies  of  drafts  of  various  documents. 
These  were  used  together  with  interviews  and  the  author’s 
own  recollections  in  order  to  reconstruct  the  process. 

An  attempt  is  made  to  note  each  source  used  in  this  recon- 
struction. Where  the  information  was  furnished  recently 
through  the  recollection  of  a participant,  as  opposed  to 
from  the  records,  the  reader  will  note  a 1979  or  1980  re- 
search note.  Any  statements  which  are  made  without  cita- 
tions are  the  recollections  of  the  author. 

The  organization  of  this  chapter  is,  for  the  most  part, 
chronological.  It  traces  the  procurements  from  the  decision 
to  contract  for  evaluations  through  to  the  signing  of  con- 
tracts and  the  debriefing  of  one  of  the  unsuccessful  bid- 
ders. For  each  section  of  the  chapter  information  is  pro- 
vided on  the  decisions  which  were  being  made,  the  people 
involved,  and  the  actual  time  frames  in  which  events  occur- 
red. Issues  which  were  raised  during  the  process  by  partici 
pants,  actions  which  were  taken,  and  problems  which  were 
encountered  are  presented.  The  emphasis  is  necessarily 
the  author’s.  Some  details  of  the  NBS  project  plan  approval 
process  which  did  not  seem  central  to  t.he  problem  have  been 


omitted . 


Several  figures  have  been  included  to  illustrate  how 
the  evaluation  teams  actually  displayed  the  ratings  of  bid- 
ders during  their  meetings.  A complete  set  of  ratings  from 
the  process  are  provided  in  the  appendix. 

Figure  IV. 1 is  provided  as  a reference  point.  It  illus 
trates  the  relationship  in  time  among  the  various  overlap- 
ping activities  which  occurred  over  the  course  of  the  pro- 
curement. It  also  identifies  the  people  or  groups  of  people 
who  were  involved  in  these  different  activities.  The  figure 
has  been  reduced  to  one  page  to  provide  the  total  time  per- 
spective at  one  glance.  A reproduction  on  a more  legible 
scale  is  provided  on  two  pages  in  the  appendix. 

This  chapter  does  not  contain  a step  by  step  guide 
on  how  to  accomplish  source  selection  by  using  Parametric 
Factor  Evaluation.  Nor  does  it  contain  explicit  descrip- 
tions of  the  various  documents  which  are  included  in  an 
RFP  using  PFE.  Such  descriptions  have  been  presented  by 
Thompson  (1976a)  in  a paper  which  is  reproduced  in  the  appen 
dix.  A summary  of  the  process  appears  in  section  II. 4 of 
this  document. 

IV . 2 Decision  to  Contract  for  Evaluation  Systems 

The  decision  to  contract  for  and  develop  evaluation 
systems  was  somewhat  diffuse.  It  occurred  over  a period 
of  about  a year  and  a half  and  involved  the  input  and  deci- 


71 


Om 

tu 

V) 


Vi  9 
T U 

u ec  e 

Vi*i  > 

X c ■>  — 

to  »•  *;  t. 


*^<0 
<aa 
> s 


au. 

a 3 
*» 
T.  n 


O *i» 

O — U ^ 

sue  U I 

S3—  rs  I 


W — u 

W ^ 

O c sa 

V — 

«r  I c c 
u * z — 

= '.l 

Ci  n a (Q 

n n It 

U U‘  u s 

U W — 
C C C (- 

o u o tu 


*»  0*^0 


0^0 

s u c • 


u — u 

O — o 

X O It  s 
3 

z s — s 

O — » X 

— X > o 

to  u:  » u 


> • 

V s 
i.  w 
o s u: 
•ox  — 
S tt  - I C H> 

M e I 
t-  — r fJ  c 
tu  c s 


; a « <•-  < 
rFSn- 
» O s «.>  < 
; £ z 9)  • 


n o — I * 

Jf  H 

f}  « UJ  I 


s w s u*'  c z 


— <r«—  X — X C 


9 eo 
E Z 
> O — 
J ^ H 
H O 


X • e 
e L.  o 

— V a*  «r» 

3 E u a 

«»—  E 

V U (»  o 
c a (-  £ 

— « o H 
o 

V u bc  • 
&'  a c s 
E — ■ 

C *•> 

— — o u 

(t  bO  It 

— «•  fc. 

X .>  U ( 

— CO; 

c r o *>- 

A« 


U ^ 

It  c 
a o 
V o 


O 'O 

E 

a X 
u o 
O f> 


f}  z 

S U 
mm  OXJ 
t-  <5  «S 
03  U 

to 

X c > 

X o — 

O H» 

■c  < 
o ^ s 
E U H 
u r to 
O 3 I- 
U.  (Q  z 

e — 

— os 

X 4^  a 
Z U <C 

l:  o 

s c S 
u o c < 
u 4^  s 
< (.  C (3 

<c  a s 
S s ck  k s u 


s u X 
s X n 
sx  — 


<e  U3  Cl 

u> 

T3»-  O 
H»  — 
O W 
3u:  ^ 
CSC 


— r 

- 19  b' 

w ^ c 
o — — 


tr  c 
one  u 
w a — k X 
16  Z ->  C>X 
— O 
<*■»  C*  < C 13 

o z c z 

tr  M 1,0  b M 
C ?•  CO  — X 

e au 


9) 

e Ti  b 

wu:  a c c 
s X o r -o 

19  h»  b « V 
c»:  — 
« o <0  — X 

to  — o s o 
CO  > s s b 
s^u  ss 


I C S M4  ^ 
I > S <0  0}  H 
I — O b < o 
mi  R C X ^ 

. O C 7 ^ 

I C R S *0  < C 
I X(T»«*—>  — 

I R 0>  H X U Q. 


•>•0  C 44  o 

b C o C3 

o V e c z — 
cj  a a<^  > 
It  c c R > e 

— 4IOO-J  — 
b T*  b Q 
C C S3-  b 

vs  c z c 

CCD  X 

C O — b 

o *■'  o a a 

4->  If  Z (t 
IT  b — *0 

b C C'  t.^ 

•j  — cj  z s 

c *a  — tr—  f— 
c — o c k-  o 
c A n c b'  u 


o z 


— « 
£ s 
X c 
a c 


« • — 


r 6 
c o c 
a X X 
CH  R 


44  S 

— K 
3 O 


C 3 • 
— It  b 
> O 


R (t 
C — 
CO  O 

O r ec 
— R C 
H-  C C 


— 

b o 
S CJ 


— US 

«•  o 

c o — 

e rt  s 

c.  xs 

E <. 
C O 
C X to 


b 

o & 
s c 


; m < l: 
r c*.  u:  s 

- C' t- 


s e o < 
— c It  - 

s — b I 

» o IT  c e 
> w c o 
•’  — o 


It  < 


C — S S I 
C IT  h-  C I 
SI  > O S - 
SU  b' s V 


' C S Z «0 

b o 

O.—  N ts 

k - *-  I- 

s-<  cs.  u O 

o s < u u 


) c -a  . 

. b c 

a L 


I a—  — — 


s — s 

03  ss  I 0}  I 


R 4'  _ 

C b c 
u so 


coo  Z I 


^ R O X 

(li  V 44  Z 

r s to 
> s o: 
C C It  s 

C C *4  44 

n c X R s 

a«.-  — O 

b <1^  £ s 

o — X — >* 

X b a 


CO 

30 


^ a — «»  44 


I 9 39,  44  4 

■ O Kmm  O i 

: c H-  lb  I 

: b C03  b I 

O *i 

R • e 

' o c e o I 

R O O U I 
I C — R 4 

> o 44  a<  I 
^ a R e 

R Z O C 4 
C CX  C ' 
U V H — 

6 44  < 

C E • CO  : 

C O b — 

44  V O 44 

44  VX  O 

— b^  V 

b a c> 

Z ^ R Z 


^ZO 
Om  O 
(9  S 
b 03  K 
Xi-4  X 
A 

A I 


JUL  SEP  OCT  HOV  DSC  JAN  FEB  HAH  APR  MAI  JUN  JUL  AUO 
197N  1975  ’976 

Figure  IV. 1 The  Activities  in  ETIP’s  Source  Selection 


72 


sions  of  several  people.  It  originated  in  1974  out  of  a 
desire  by  the  director  of  ETIP  for  more  explicit  integration 
of  evaluations  into  the  programs'  activities  (ETIP  Director, 
Note  1 ) . 

The  major  part  of  the  program  effort  to  develop  an 
evaluation  capability  began  in  early  September  of  1975  with 
the  arrival  at  ETIP  of  the  person  who  became  the  Contracting 
Officer's  Technical  Representative  for  the  two  evaluation 
systems  contracts.  The  COTR  was  brought  into  ETIP  on  an 
Intergovernmental  Personnel  Act  appointment.  He  was  to 
provide  evaluations  of  the  program's  experiments  which  woul'd 
produce  information  for  use  in  an  overall  ETIP  program  re- 
view scheduled  for  1980  (COTR,  Note  2). 

The  Director  of  ETIP  decided  that  money  would  be  obli- 
gated to  contract  for  evaluation  services  (COTR,  Note  3). 
Since  the  COTR  had  no  staff,  there  was  never  any  serious 
consideration  of  doing  the  work  in-house  (COTR,  Note  4). 

It  was  evident  to  the  COTR  that  if  the  program  were 
to  continue  to  exist  beyond  its  1980  program  review  it  would 
need  some  type  of  on-going  capability  to  evaluate  its  future 
experiments,  as  well  as  its  past  and  current  experiments. 

He  wished  to  develop  an  evaluation  capability  based  on  gen- 
eral evaluation  designs  which  would  build  upon  experience 

1 

This  person  will  be  referred  to  as  the  COTR  throughout 
this  document. 


and  not  require  having  to  start  over  by  developing  designs, 
data  collection  plans,  and  data  analysis  plans  for  each 
new  experiment.  He  also  wanted  those  capabilities  to  be 
developed  within  the  government  as  opposed  to  having  to 
depend  largely  on  one  or  more  contractors  over  an  extended 
period  of  time  for  evaluation  expertise.  In  order  to  do 
this  he  realized  that  he  needed  contractors  who  were  capable 
of  working  with  ETIP  over  a period  of  time  to  design,  de- 
velop and  transfer  such  an  evaluation  capability  (COTR, 

Note  5 , Note  6 ) . 

The  use  of  the  term  system  and  the  incorporation  of 
other  system  concepts  did  not  become  part  of  the  projects 
until  work  on  preparing  the  RFP  began  (COTR,  Note  7).  In 
this  earlier  period  of  the  development  of  the  project  the 
objectives  were  described  in  terms  of  developing  designs, 
capabilities,  and  doing  specific  experiments  (COTR,  Note 
8).  These  were  all  incorporated  into  what  were  later  de- 
scribed as  the  systems. 

Records  indicate  that  the  COTR  started  writing  a plan 
for  a project  to  evaluate  the  commercial  impacts  of  ETIP’s 
procurement  policy  experiments  as  soon  as  he  arrived  at 
ETIP  (COTR,  Note  9).  The  administration  of  NBS  required 
ETIP  to  prepare  and  submit,  for  its  review  and  approval, 
a plan  for  each  project  which  ’would  result  in  the  obligation 
of  more  than  $10,000.  At  this  time  in  the  history  of  the 
program  the  approval  process  for  ETIP  project  plans  was 


quite  extensive.  Four  people,  ending  with  the  Acting  Direc- 
tor of  NBS,  were  required  to  sign  off  on  the  plans  and  sever 
al  others  reviewed  them  before  they  were  submitted  to  those 
who  were  required  to  sign. 

The  COTR  used  the  requirement  to  prepare  the  project 

plan  as  a method  for  familiarizing  himself  with  the  problem 

area  with  which  he  would  be  dealing  in  his  new  job.  He 

spent  several  months  talking  with  other  staff  at  both  ETIP 

2 

and  at  the  Program  Administrative  Agencies  (PAAs)  and  read- 
ing existing  program  publications  in  order  to  determine 
potential  topics  and  methods  for  evaluation.  When  he  com- 
pleted the  first  draft  of  his  plan  he  circulated  it  to  the 
ETIP  staff  and  used  their  comments  to  revise  it  into  a for- 
mat which  the  ETIP  reviewers  thought  would  be  more  accept- 
able for  the  NBS  review  process.  The  completed  commercial 
impact  evaluation  plan  was  then  used  as  the  basis  for  writ- 
ing a plan  to  evaluate  the  agency  impacts  of  the  same  experi 
ments  (COTR,  Note  10).  This  entire  process  of  writing, 
review,  and  approval  lasted  until  March,  1976  (NBS  Acting 
Director,  Note  11). 

2 

Program  Administrative  Agencies  (PAAs)  is  the  term 
ETIP  used  to  refer  to  the  organizations  having  responsibil- 
ity and  authority  for  decisions  to  introduce  experimental 
interventions.  In  this  case  the  Federal  Supply  Service 
(FSS),  the  Veterans’  Administration,  and  state  and  local 
procurement  agencies  (or  their  consortia). 


An  important  uncertainty  which  existed  was  whether 
the  required  capabilities  existed  in  any  contractor  organiza 
tions  (Thompson,  1976a).  No  one  in  ETIP  was  aware  of  any 
instances  where  the  government  had  successfully  procured 
a similar  evaluation  capability.  This  was  particularly 
troublesome  since  contracts  for  single  evaluations  had  such 
a poor  track  record  (see  section  II. 2)  and  few  people  in 
the  evaluation  industry  appeared  to  have  the  system  design 
and  development  experience  which  seemed  necessary  to  provide 
for  the  systems  aspects  of  the  project.  In  addition  there 
were  problems  of  attracting  bids  by  any  firms  which  might 
have  the  required  capabilities  and  then  evaluating  the  propo 
sals  to  determine  the  degree  to  which  the  capability  or 
potential  capability  did,  in  fact,  exist.  The  procurement 
process  would  have  to  address  these  problems. 

IV.  3 Preparing  the  Request  for  Proposals 

The  COTR  wanted  to  develop  a RFP  in  parallel  with  his 
project  plans.  However,  he  realized  that  he  would  be  unable 
to  do  both  effectively  within  a time  frame  he  felt  satisfied 
with  (COTR,  Note  12).  In  early  October  he  asked  Charles 
Thompson  to  write  an  RFP  for  a contract  to  evaluate  the 
commercial  impacts  of  the  ETIP  experiments  with  state  and 

Thompson  had  been  familiar  with  ETIP  since  the  summer 
of  197^.  He  had  been  associated  with  the  program  as  a spe- 
cial employee  reporting  to  the  Director  since  September, 
1975,  while  remaining  at  Northwestern  University. 


local  procurement  organizations  (COTR,  Note  13)*  He  wrote 
to  Thompson  and  sent  him  some  background  materials  from 
a prior  unsuccessful  attempt  at  awarding  a similar  contract 
(COTR,  Note  14).  Thompson's  background  in  both  evaluation 
and  systems  management  was  an  indication  to  the  COTR  that 
he  might  be  able  to  provide  the  help  which  was  needed  (COTR 
Note  15). 

At  this  time  the  COTR  had  not  thought  through  the  num- 
ber or  structure  of  the  contracts  which  would  be  awarded 
when  he  received  NBS  authorization  on  the  two  project  plans 
he  was  writing.  He  knew  that  at  least  one  RFP  was  needed 
and  that  he  could  use  that  as  a model  for  other  RFPs  (COTR, 
Note  16 ) . 

The  COTR  and  Thompson  first  met  to  discuss  the  RFP 
on  October  20,  1975.  They  began  talking  about  some  of  the 
issues  which  would  be  important  in  the  proposed  procurement 
Thompson  felt  it  was  important  that  the  RFP  be  successful 
in  obtaining  proposals  and  result  in  a contract  award. 
Specifically,  he  was  concerned  that  go'od  companies  would 
begin  to  turn  down  (not  bid)  or  ignore  ETIP  work  if  the 
procurement  process  ended  up  rejecting  all  bids  (COTR,  Note 
17).  This  concern  was  particularly  salient  since  the  prior 
RFP  for  evaluation  of  ETIP's  procurement  experiments  had 
been  cancelled  that  same  year  after  the  proposals  had  been 


evaluated. 


other  items  discussed  during  the  October  20  meeting 
included ; 

o how  to  provide  for  an  information  exchange 
through  the  procurement  process  which  would 
be  effective  in  satisfying  the  needs  of  both 
prospective  bidders  and  the  proposal  evalua- 
tion panel.  That  is,  the  needs  of  prospec- 
tive bidders  to  determine  how  to  match  his 
proposal  with  the  requirements  under  the 
contract  and  the  needs  of  the  evaluation 
panel  to  be  able  to  determine  the  bidder’s 
understanding  of  the  project  and  his  capabil- 
ity to  perform  effectively; 

o how  to  assess  the  importance  of  the  project 
to  top  management  of  a bidder’s  firm  in  order 
to  be  confident  that  the  necessary  resources 
will  be  devoted  throughout  the  contract; 

o how  to  make  ETIP’s  priorities  clear  in  the 
RFP;  and 

o how  to  determine  whether  a bidder  will  be 
able  to  meet  the  multiple  objectives  of  var- 
ious key  people  involved  in  the  projects 
being  evaluated  (COTR,  Note  18). 

Thompson  decided  to  use  the  Parametric  Factor  Evalua- 
tion approach  (PFE)  when  he  recognized  the  apparent  similar- 
ities between  the  procurement  of  an  evaluation  capability 
which  the  COTR  desired  for  ETIP  and  the  procurement  of  the 
military  hardware  sub-systems,  for  which  the  technique  was 
developed  (Thompson,  1976a).  He  described,  in  a subsequent 
paper,  three  similarities  which  occurred  to  him  in  making 
this  decision. 

First,  the  evaluation  requirements  were  not  only 
constantly  changing  because  of  the  on-going  pro- 
gress in  the  initiation  and  carrying-out  of  pro- 
curement experiments  but  also  included  a wide 


variety  of  objectives  ranging  from  very  specific 
to  very  broad,  and  from  intermediate  to  long- 
range.  Second,  ETIP  had,  with  the  exception  of 
the  writer  [Thompson],  little  or  no  experience 
in  systems  procurement,  and  neither  the  time  nor 
personnel  to  accomplish  a problem  or  concept  defi- 
nition design.  Finally,  it  did  not  appear  that 
the  "evaluation  industry"  had  any  significant 
present  capability  to  design  or  manage  an  "evalua- 
tion system"  to  meet  ETIP  requirements  (Thompson, 

1976a) . 

The  situation  was  one  in  which  high  mutual  uncertainty 
existed  between  ETIP  and  its  potential  bidders.  It  was 
unclear  what  the  specific  nature  of  the  final  evaluation 
systems  would  be.  Such  basic  issues  as  how  many  systems 
there  would  be  and  who  would  own  them  were  still  unresolved. 
It  was  also  uncertain  what  specific  steps  would  need  to 
be  taken  in  the  development  process.  The  nature  of  the 
situation  seemed  to  suggest  the  need  for  a statement  of 
work  which  would  allow  for  the  uncertainty  rather  than  one 
which  specified  numerous  details  (COTR,  Note  19). 

For  ETIP  a flexible  statement  of  work  presented 
risks  that  the  contract  would  not  result  in  the  de- 
sired systems.  Therefore,  it  was  important  to  be 
able  to  determine  the  degree  to  which  it  was  within 
each  bidder's  interest  as  well  as  their  capability 
to  develop  the  systems.  The  RFP  would  have  to  provide  the 
structure  for  making  these  assessments. 

Thompson’s  role  was  to  work  with  the  COTR  in  developing 
the  concepts  of  what  ETIP  would  be  procuring  and  to  write 


the  RFP  based  on  this.  It  was  Thompson  who  introduced  the 
concept  of  evaluation  systems  as  the  method  for  procuring 
the  on-going  evaluation  capability  which  the  COTR  desired 
(Thompson,  Note  20;  & COTR,  Note  21).  Thompson’s  role  also 
included  participation  in  the  evaluation  and  advising  the 
COTR  on  all  other  aspects  of  using  the  PFE  process. 

The  COTR  and  Thompson  met  for  a second  time  on  November 
3 and  4 to  further  discuss  issues  of  concern  related  to 
the  RFP  and  the  eventual  contracts.  These  meetings  served 
a mutual  education  purpose.  Thompson  learned  the  COTRs’ 
and  others’  desires  and  concerns  which  he  used  in  writing 
the  sub-factors.  He  also  worked  to  make  the  COTR  aware 
of  some  of  the  important  aspects  of  system  procurement  (COTR, 
Note  22) . 

ft 

On  November  5,  Thompson  sent  a letter  to  the  COTR  to- 
gether with  some  key  documents  from  the  prior  Air  Force 
uses  of  the  PFE  process.  The  purpose  of  sending  these  docu- 
ments was  to  begin  to  provide  the  COTR  with  perspective 
and  background  on  the  process  and  to  build  his  confidence 
in  the  process  by  showing  that  it  was  not  untested  (Thomp- 
son, Note  23). 

On  November  17,  Thompson  met  with  the  COTR  and  describ- 
ed a list  of  evaluation  factors.  There  were  two  management 
factors  and  five  technical  factors.  The  management  factors 


were : 


o management  style  of  the  contractor  in  relat- 
ing to  ETIP  and 

o internal  management  of  the  project. 

The  technical  factors  were: 

o understanding  and  conceptualization  of  the 
problem, 

o collecting  background  on  sites, 
o research  design, 
o collecting  data,  and 
o analyzing  data  (COTR,  Note  24). 

The  management  factors  came  from  a scaling  down  of  the  lar- 
ger number  of  management  factors  used  by  the  Air  Force  (Thomp- 
son, Note  25).  Thompson  derived  the  technical  factors  based 
on  his  knowledge  of  what  ETIP  was  buying,  his  reading  of 
ETIP  background  documents,  and  his  discussion  with  ETIP 
staff  (COTR,  Note  26;  Thompson,  Note  27).  This  background 
suggested  issues  for  the  sub-factors  and,  at  a higher  level 
of  aggregation,  the  factors.  Determining  the  factors  and 
sub-factors  was  an  iterative  process  rather  than  sequential. 
Additional  sub-factors  were  added  after  the  factor  groupings 
were  determined  (Thompson,  Note  28). 

Following  his  discussions  with  the  COTR,  Thompson  wrote 

* 

a draft  of  the  document  entitled  "Proposal  Evaluation  Fac- 
tors" (Thompson,  Note  29).  This  was  the  listing  of  all 
of  the  factors  and  sub-factors  which  would  be  used  as  the 
structure  for  both  the  preparation  and  evaluation  of  propo- 


4 

sals.  The  structure  would  also  be  used  for  organizing 
the  ’’Statement  of  Work,”  ’’Schedule,”  and  ’’Scope  of  Effort” 
documents  which  remained  to  be  written.  Thompson  worked 
on  writing  these  and  other  documents  which  would  be  given 
to  the  contract  officer  to  be  included  in  the  solicitation 
document.  By  February  11,  he  had  completed  drafts  of  the 
following : 

o ’’Introductory  Note,” 
o ’’Definitions,” 
o ’’Statement  of  Work,” 

o ’’Statement  of  Work  (with  illustrative  sub-items),” 
o Schedule  (overall),” 
o ’’Schedule  (Phase  D,” 
o ’’Proposal  Preparation  Procedure,” 

0 ’’Proposal  Evaluation  Factors,” 

0 ’’Proposal  Evaluation  Factors  (with  illustrative 
sub-questions) , ” 

o ’’Scope  of  Effort,”  and 

o ”Factor/Sub-factor  worksheet”  (Thompson,  Note  30). 

4 

There  were  seven  factors  and  33  sub-factors  in  the 
final  version. 

5 

The  form  and  in  some  cases  the  content  of  these  docu- 
ments followed  that  of  prior  Air  Force  procurements  using 
PFE.  Texts  of  the  ’’Proposal  Preparation  Procedure”  and 
’’Scope  of  Effort”  follow  closely  those  used  by  the  Air  Force. 
The  final  version  of  each  of  these  documents  may  be  found 
in  the  appendix. 


As  drafts  of  documents  and  information  about  the  PFE 
process  became  available  at  ETIP,  questions  and  concerns 
about  the  procedure  began  to  arise.  An  issue  arose  during 
discussion  among  some  of  the  ETIP  staff  which  related  to 
the  proposed  ”non-blind"  evaluation  of  proposals.  Some 
staff  members  felt  strongly  that  the  proposals  should  be 
evaluated  "in  the  blind."  This  was  the  technique  which 
ETIP  had  been  using  in  its  other  procurements.  That  is, 
proposals  were  submitted  with  no  identification  of  the  bid- 
ders. The  evaluation  panel  based  its  ratings  solely  on 
the  proposed  approach  in  the  document  without  knowing  who' 
the  bidders  were.  It  was  felt  that  this  would  produce  a 
more  objective  evaluation  and  that  the  panel  could  not  be 
accused  of  being  biased  (COTR,  Note  3 1 ) • By  contrast,  PFE 
requires  explicit  identification  of  the  bidder  and  many 
of  the  illustrative  questions  for  the  sub-factors  inquired 
into  both  prior  and  anticipated  future  work  of  the  firm 
and  the  staff  which  would  be  assigned  to  the  proposed  pro- 
ject. Proposal  evaluation  panel  members  having  any  informa 
tion  which  would  bias  their  ratings  either  for  or  against 
a specific  bidder  are  required  to  make  this  known  to  the 
others  during  the  factor  team  meetings.  This  issue  was 
resolved  by  the  director  of  ETIP  after  Thompson  argued  that 
it  would  be  impossible  to  evaluate  the  factors  without  know 
ing  the  identity  of  the  bidders  and  that  it  would  be  impos- 
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sible  to  keep  the  identity  of  the  bidders  a secret  even 
if  it  was  blind.  The  director  agreed  to  a non-blind  evalua- 
tion (Thompson,  Note  32). 

When  strategy  for  the  scope  and  number  of  evaluation 
contracts  was  discussed  in  December,  it  was  thought  that 
four  contracts  would  be  awarded  (Figure  IV. 2).  There  would 
be  one  to  develop  a system  to  evaluate  the  commercial  im- 
pacts of  federal,  state  and  local  procurement  experiments, 
one  each  to  develop  systems  to  evaluate  the  agency  impacts 
of  a)  federal  and  b)  state  and  local  experiments  and  a fourth 
to  evaluate  both  the  agency  and  commercial  impacts  of  experi- 
ments with  the  Veterans  Administration  (COTR,  Note  33;  Note 
34).  The  specific  program  decision  to  attempt  to  award 
two  contracts  for  the  design  and  development  of  one  or  more 
evaluation  systems  occured  on  January  16,  1976,  when  this 
issue  was  discussed  in  a meeting  among  the  Director  of  ETIP, 
the  COTR,  and  Thompson.  The  COTR  was  instructed  by  the 
Director  to  award  two  contracts,  one  for  each  of  the  project 
plans  he  was  writing,  and  to  do  so  by  June  30  (the  end  of 
the  fiscal  year).  Thompson  argued  that  ETIP  would  be  unable 
to  handle  the  evaluation  process  for  more  than  two  contracts 
(Thompson,  Note  35).  The  Director  felt  that  any  more  than 
two  contracts  would  be  too  much  for  the  COTR  to  manage  effec- 
tively. 

A budget  needed  to  be  established  for  each  contract 
in  order  to  indicate  the  size  of  the  contracts  to  prospec- 
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FIGURE  IV. 2 


INITIAL  CONCEPT  OF  THE  STRUCTURE  OF  THE 
EVALUATION  CONTRACTS 


tive  bidders.  Due  to  uncertainties,  anticipated  and  unan- 
ticipated, it  was  not  feasible  to  determine  what  the  even- 
tual costs  would  be.  The  budgets  needed  to  be  attractive 
to  bidders,  be  responsive  to  the  pressures  to  obligate  pro- 
gram funds  within  the  fiscal  budget  cycle,  and  allow  the 
government  reasonable  control  over  the  development  expendi- 
tures. If  the  budget  were  too  small  there  was  thought  to 
be  a good  chance  that  organizations  would  not  invest  in 
responding.  The  program  was  under  pressure  from  the  Depart- 
ment to  spend  money  so  as  not  to  have  a budget  carryover 
at  the  end  of  the  fiscal  year  (COTR,  Note  36).  Counterbal- 
ancing pressures  existed  from  the  Program  Office  at  NBS 
to  control  costs  (COTR,  Note  37).  When  the  Director  of 
ETIP  decided  to  have  two  contracts,  he  also  decided  that 
they  would  be  phased  contracts  and  that  the  first  two  phases 
of  each  would  be  funded  at  approximately  one  million  dollars 
(Thompson,  Note  38).  This  presented  projects  of  reasonable 
size  to  attract  bids,  allowed  the  program  to  obligate  its 
budget,  was  thought  to  be  an  acceptable  number  of  contracts 
for  the  COTR  to  manage.  The  phasing  of  the  contracts  pro- 
vided a formal  checkpoint  at  the  end  of  the  first  year  to 
determine  whether  the  money  was  being  spent  effectively 
(Thompson,  Note  39). 

With  drafts  of  the  technical  aspects  of  the  RFP  nearing 
completion  and  the  assignment  of  a June  30  award  deadline. 


it  was  important  to  contact  the  Department  of  Commerce’s 
contracting  office  in  order  to  begin  the  solicitation  pro- 
cess . 


IV. 4 Involving  the  Contracting  Office 

The  contracting  office  had  not  yet  been  involved  in 
any  of  the  preparations  for  this  procurement.  It  was  now 
important  to  determine  whether  the  PFE  approach  appeared 
to  be  acceptable  in  general  and  to  begin  working  out  the 
specifics  needed  in  the  process  (Thompson,  Note  40). 

Thompson  had  been  having  arguments  with  the  director 
of  ETIP  and  another  senior  staff  member  over  the  statement 
of  work.  The  statement  of  work  followed  the  structure  of 
the  factors  and  was  very  broad.  It  indicated  the  categories 
of  work  to  be  done  but  was  not  specific  about  interim  de- 
liverables. The  senior  staff  member  and  the  director  wanted 
it  to  be  more  detailed  (Thompson,  Note  41).  The  ETIP  admini- 
strative officer  set  up  a meeting  with  people  in  the  con- 
tracting office  to  try  to  resolve  whether  the  PFE  process 
and  the  flexible  statement  of  work  would  be  acceptable. 

She  arranged  to  meet  with  the  contracting  officer  and  con- 
tract negotiator  who  she  knew  wpuld  be  most  receptive  to 
helping  ETIP  determine  whether  the  process  would  be  legal 
and  workable  (ETIP  Administrative  Officer,  Note  42). 


The  meeting  was  held  on  January  27  and  was  attended 
by  Thompson,  the  COTR,  the  ETIP  administrative  officer, 
the  contracting  officer,  and  the  contract  negotiator.  The 
contracting  officer  was  very  receptive  to  the  PFE  approach. 

He  had  worked  with  the  Maritime  Administration  and  saw  the 
approach  as  similar  to  those  used  to  purchase  ships  (ETIP 
Administrative  Officer,  Note  43).  After  it  was  agreed  that 
PFE  process  could  be  used,  some  of  the  specific  steps  were 
discussed.  Since  the  PFE  process  was  a departure  from  most 
other  Department  of  Commerce  procurement  methods,  it  was 
agreed  to  brief  the  bidders  twice.  This  would  give  the 
bidders  an  opportunity  to  familiarize  themselves  with  the 
requirements  and  to  ask  questions.  It  was  felt  that  an 
open  exchange  of  information  would  enable  bidders  to  see 
that  ETIP  had  no  sole  source  position  and  would  help  bidders 
to  determine  whether  they  wanted  to  respond  (ETIP  Administra- 
tive Officer,  Note  44).  Scheduling  of  the  two  briefings 
and  of  the  evaluation  process  were  discussed  and  the  con- 
tracting officer  assigned  the  responsibility  for  the  next 
steps  in  the  procurement  to  the  contract  negotiator  (COTR, 
Note  45).  It  was  agreed  that  the  final  RFP  would  be  issued 
prior  to  the  second  bidder’s  briefing. 

Though  the  contracting  officer  agreed  that  the  broad 
statement  of  work  was  sufficient,  the  director  of  ETIP  still 
wanted  additional  detail  specified.  To  meet  this  require- 


88 


merit,  Thompson  wrote  a second  statement  of  work  which  con- 
tained detailed  illustrative  sub-items  (Thompson,  Note  46). 
This  was  included  in  the  RFP , together  with  the  actual  state- 
ment of  work. 

The  schedule  which  was  worked  out  with  the  contracting 
office  indicated  that  it  was  necessary  to  begin  briefing 
prospective  bidders  in  mid-February  in  order  to  be  able 
to  meet  the  June  30,  award  deadline.  The  announcement  of 
the  solicitation  had  to  be  made  immediately. 

IV. 5 Involving  NBS  Management 

On  January  29,  the  director  of  ETIP  sent  a memo  to 
the  Acting  Director  of  the  National  Bureau  of  Standards 
indicating  ETIP’s  intention  to  award  two  parallel  contracts 
for  approximately  one  million  dollars  each.  With  the  memo 
he  sent  a copy  of  the  announcement  of  the  procurement  and 
the  schedule  of  briefings  which  were  being  sent  to  the  Com- 
merce Business  Daily.  He  indicated  that  this  announcement 
did  not  obligate  ETIP  to  make  an  award  but  that  the  impor- 
tance of  beginning  to  educate  contractors  about  the  procure- 
ment required  publishing  the  notice  before  NBS  approval 
of  the  project  plans.  He  promised  that  the  plans  would 
be  circulated  for  approval  prior  to  the  pre-bidders  briefing 
of  February  26th  (ETIP  Director,  Note  47). 


Issuing  an  announcement  of  intention  to  let  contracts 
prior  to  NBS  review  of  project  plans  was  counter  to  normal 
procedure.  However,  this  was  thought  to  be  necessary  in 
order  to  achieve  the  June  30  award  target.  It  is  likely 
that  this  action  combined  with  the  size  of  the  proposed 
contracts  (large  in  comparison  to  most  other  NBS  contracts) 
created  a great  deal  of  anxiety  among  the  NBS  administra- 
tion. The  program  office  wanted  assurance  that  the  output 
of  the  project  would  support  commitments  which  NBS  had  made 
for  future  evaluations  of  the  ETIP  program  and  that  no  im- 
plicit future  funding  commitments  were  being  made  (COTR, 

Note  48,  Note  49,  Note  50).  The  COTR  and  others  at  ETIP 
met  with  the  program  office  staff  throughout  the  project 
plan  review  process  to  brief  them  and  answer  their  questions 
about  the  projects.  On  March  31,  1976  the  Acting  Director 
of  NBS  approved  the  project  plans  (NBS  Acting  Director, 

Note  51).  This  cleared  the  way  for  the  award  of  contracts 
following  completion  of  the  evaluation  of  proposals. 

IV. 6 Involving  the  Program  Administrative  Agencies 

It  was  thought  to  be  important  to  use  the  proposal 
evaluation  process  as  a method  for  involving  and  educating 
people  who  would  be  involved  in  the  project.  Participation 
would  engage  people  in  a dialogue  about  what  ETIP  was  trying 
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to  do  with  the  project  and  some  of  the  problems  involved 
in  doing  it.  This  was  particularly  important  to  staff  of 
the  PAAs. 

By  February  11,  drafts  of  most  of  the  technioal  docu- 
ments for  the  RFP  were  completed  and  ready  for  review  by 
the  staffs  of  ETIP  and  the  Program  Administrative  Agencies. 
Since  people  in  the  PAAs  would  be  working  with  the  contrac- 
tors, it  was  important  that  they  be  involved  in  the  solici- 
tation process.  Normally,  as  part  of  the  PFS  process,  their 
aid  would  have  been  requested  in  determining  the  sub-factors 
and  factors.  However  the  time  pressures  of  this  procurement 
prevented  this  level  of  involvement  (COTR,  Mote  52).  All 
that  could  be  accomplished  now  would  be  to  obtain  their 
review  of  the  various  draft  documents  to  insert  anything 
which  may  have  been  overlooked. 

The  only  Program  Administrative  Agency  to  be  involved 
actively  in  the  review  process  was  the  Federal  Supply  Service 
(FSS).  The  National  Institute  for  Government  Procurement 
and  the  National  Association  of  State  Purchasing  Officials, 
which  coordinated  ETIP’s  local  and  state  purchasing  experi- 
ments, could  not  be  involved  in  the  review  of  the  drafts 
since  people  who  were  not  federal  government  employees  were 
not  permitted  to  see  the  RFP  prior  to  its  release  (COTR, 

Note  53 )•  ETIP's  work  with  the  Veterans’  Administration 
was  in  a state  of  flux.  The  primary  contact  point  had  re- 
cently changed  and  there  was  some  disagreement  and 


confusion  regarding  the  role  that  the  contracts  would  play 
in  this  work.  The  VA  was  represented  in  the  initial  review 
of  the  RFP  but  did  not  participate  in  the  eventual  evalua- 
tion of  proposals. 

The  primary  ETIP  contact  at  FSS  had  recently  assumed 
his  position  there  as  Director  of  their  Experimental  Technol- 
ogy Division  after  having  been  chief  of  ETIP's  procurement 
program.  He  was  familiar  with  ETIP’s  plans  for  evaluation 
and  was  quite  concerned  about  the  large  size  of  the  effort 
and  prospect  of  the  resulting  contractors  working  in  his 
agency  (COTR,  Note  54).  When  the  RFP  documents  were  sent 
to  FSS  for  review  he  and  his  staff  made  extensive  comments 
on  both  the  process  and  the  specific  sub-factors.  Their 
concerns  included  the  following; 

o that  the  amount  of  control  and  responsibility 
given  to  the  contractor  seemed  excessive. 

They  felt  that  many  of  the  sub-factor  issues 
which  the  contractors  were  being  asked  to 
respond  to  were  ETIP’s  responsibility,  e.g., 
rationalizing  multiple  and  conflicting  objec- 
tives of  ETIP,  telling  them  what  information 
ETIP  will  need  to  review  the  project, 

o the  work  to  be  performed  overlapped  with 
existing  projects, 

o the  amount  of  money  being  allocated  was  "absurd" 
for  the  work  they  felt  needed  to  be  done, 

o much  of  the  information  provided  to  bidders 
was  unnecessary,  particularly  that  informa- 
tion which  gave  the  history  of  the  PFE  process 
and  the  details  of  how  the  proposals  would 
be  evaluated, 

that  the  organization  of  the  RFP  was  bad 
and  some  paragraphs  were  confusing. 


o 


0 that  it  was  unclear  why  two  statements  of 
work  were  included  (one  with  illustrative 
sub-items) , 

o that  it  would  take  the  panel  two  to  three 
months  to  evaluate  the  proposals  which  would 
be  received  in  response  to  the  RFP, 

o many  of  the  sub-factors  were  not  useful, 
especially  those  on  management  and  report- 
ing, and 

o that  the  RFP  would  scare  off  good  bidders 
who  would  find  it  confusing, 

o that  successful  contractors  would  provide 
material  to  the  PAAs  which  would  be  unusable 
(FSS  representatives.  Note  55). 

These  issues  continued  to  be  a source  of  concern  to  some 

FSS  staff  throughout  the  solicitation  process  (FSS  represen 

tative , Note  56 ) . 


IV. 7 Involving  the  Prospective  Bidders 

One  objective  of  the  PFE  process  is  to  provide  prospec 
tive  bidders  with  as  much  information,  openly  and  explic- 
itly, as  is  possible  (Thompson,  1976a).  One  method  used 
for  doing  this  was  through  the  use  of  two  bidders’  brief- 
ings. The  first  was  to  give  bidders  an  overview  of  ETIP, 
a presentation  of  the  objectives  of  the  procurement,  and 
to  allow  for  extensive  questions  and  answers.  The  second 
was  to  allow  for  further  questionning  after  bidders  had 
an  opportunity  to  review  both  oral  and  written  materials 
(Thompson,  1976). 


On  February  17th,  the  same  day  that  the  COTRs ’ two 
project  plans  were  sent  to  the  NBS  program  office  for  approv 
al,  notice  of  the  solicitation  appeared  in  the  Commerce 
Business  Daily  (CBD).  The  announcement  described  the  gener- 
al subject  nature,  level  of  effort,  and  time  frame  of  the 
intended  contracts  and  announced  the  schedule  and  procedure 
for  distribution  of  preliminary  documents,  briefings,  and 
responses  (’’Design  and  develop  evaluation  systems  for  experi 
ments...,”  1976  & Figure  IV. 3).  Individuals  who  wrote  or 
stopped  by  the  contracting  office  and  those  who  were  on 
the  bidder’s  list^  were  furnished  copies  of  the  preliminary 
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RFP  prior  to  the  pre-bidders  briefing.  The  CBD  announce- 
ment was  the  first  notification  of  the  RFP  to  be  issued 
to  the  contracting  community.  The  only  other  prior  related 
information  was  from  conversations  which  the  COTR  (and  prob- 
ably others)  had  with  contractor  organizations  to  describe 
ETIP,  its  work  and  the  general  direction  being  taken  in 
the  evaluation  of  ETIP  experiments.  By  doing  this  the  COTR 
hoped  to  create  an  awareness  of  the  program’s  activities 
so  that  when  CBD  announcements  were  published  people  would 
take  the  time  to  consider  whether  to  bid  (COTR,  Note  57). 

£ 

°The  bidders  list  was  composed  of  names  of  individuals 
and  firms  known  to  ETIP  and  thought  to  have  a potential 
interest  in  learning  about  the  project. 
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The  final  version  of  the  RFP  was  to  be  issued  prior  to 
the  second  briefing. 


H— DESIGN  AND  DEVELOP  EVALUATION  SYSTEM  FOR  PROCUREMENT 
EXPERIMENTS  on  behalf  of  ETIP  (Experimental  Technology  Incen 
fives  Program).  Obtain  evaluation  system  to  measure  the 
agency  impact  and  commercial  impact  of  a series  of  procure- 
ment experiments  being  conducted  in  cooperation  with  specifi 
government  purchasing  agencies.  Two  contracts,  one  for 
the  agency  impact  and  one  for  the  commercial  impact,  each 
with  a level  of  effort  of  approx.  100  man-months  are  to 
be  awarded.  Each  contract  will  be  for  a three  year  period, 
performance  in  three  phases.  Funding  for  the  first  two 
phases  of  each  contrac't  is  presently  available.  Because 
of  the  complexity  of  the  program,  and  the  importance  of 
selecting  contractors  qualified  to  provide  comprehensive 
program  management  to  design,  develop  and  test  an  evalua- 
tion system  and  to  carry  out  specific  evaluations,  informa- 
tion for  prospective  offerors  will  be  extensive  and  will 
be  furnished  in  the  following  steps:  a)  o/a  12  Feb  76  a 

preliminary  description  of  both  solicitations  will  be  made 
available  upon  written  request  from  prospective  offerors; 
b)  a preliminary  briefing  will  be  held  at  10  o’clock  am, 
local  Washington,  DC  time,  26  Feb  76,  in  Rm  6802  at  the 
Dept,  of  Commerce,  l4th  and  Constitution  Ave. , NW.  Washing- 
ton, DC,  to  provide  a full  description  of  these  requirements 
of  the  solicitations,  and  to  answer  questions;  a copy  of 
the  solicitation  will  be  furnished  to  each  firm  having  a 
r.epresentat ive  at  the  preliminary  briefing,  provided  the 
firm  has  furnished  the  Government  prior  written  notice  of 
attendahce;  firms  in  attendance  without  prior  notification 
will  be  mailed  a copy  of  the  solicitation  as  soon  as  pos- 
sible thereafter.  Other  firms  will  be  provided  a copy  as 
soon  as  possible  upon  receipt  of  a written  request,  accom- 
panied by  a description  of  the  firm’s  capability  to  perform 
the  type  of  work  required;  c)  the  formal  pre-proposal  con- 
ference will  be  held  at  11:00  am,  local  Washington,  DC  time, 
on  18  Mar  76  in  Rm  4833  at  the  Dept,  of  Commerce  (same  ad- 
dress as  preliminary  briefing)  to  provide  additional  infor- 
mation and  answer  questions.  Proposals  in  response  to  Sol. 
6-35756  shall  be  due  o/a  15  Apr  76.  See  Notes  24  and  80 
(P042) 
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THE  COMMERCE  BUSINESS  DAILY  ANNOUNCEMENT 


Following  the  issuing  of  the  CBD  announcement  and  com- 


pletion of  his  project  plans  the  COTR  turned  his  attention 
to  planning  the  pre-bidder’s  conference.  A dry  run  for 
the  pre-bidders  briefing  was  held  on  February  25th  and  was 
attended  by  the  contract  negotiator,  several  ETIP  staff 
and  staff  from  VA  and  NIGP.  The  COTR’s  objective  for  the 
first  briefing  was  to  communicate  what  it  was  ETIP  was  try- 
ing to  buy,  to  generate  enthusiasm  for  bidding,  to  give 
prospective  bidders  an  opportunity  to  ask  questions,  and 
to  obtain  feedback  on  the  preliminary  RFP  which  would  allow 
for  clarifications  prior  to  the  next  briefing  (COTR,  Note 
58).  He  wanted  to  convey  that  ETIP  was  interested  in  buying 
both  systems  and  specific  evaluations  and  that  methodolo- 
gists were  needed  who  could  integrate  the  different  type 
of  specialists  required  into  an  effective  team.  He  also 
felt  it  important  to  convey  the  message  that  ETIP  was  look- 
ing for  contractors  who  would  assume  a lead  position  in 
doing  the  work  while  being  sensitive  to  the  needs  of  ETIP 
and  its  PAAs.  Finally  he  wanted  to  convey  to  the  bidders 
that  ETIP  did  not  have  a sole  source  position  on  the  con- 
tract and  to  provide  as  much  information  as  possible  about 
the  program  and  the  proposal  evaluation  process  so  that 
firms  could  make  informed  decisions  about  whether  to  bid 
(COTR,  Note  59;  Note  60). 


The  pre-bidders  briefing  was  attended  by  representa- 
tives of  47  organizations  ("Attendees  at  Briefing,"  Note 
6l).  It  was  opened  with  some  remarks  by  the  contract  negoti- 
ator. He  indicated  that  though  the  solicitation  document 
itself  would  not  be  available  until  a later  date,  the  infor- 
mation which  was  already  furnished  to  prospective  bidders 
constituted  the  technical  aspects  of  the  solicitation. 

The  document  itself  would  include  some  additional  instruc- 
tions and  conditions,  special  provisions  applicable  to  the 
contract,  and  other  boiler-plate  provisions  (Transcript, 

Note  62).  He  also  indicated  that  the  two  contracts  would 
be  awarded  to  separate  firms  and  he  corrected  an  error  which 

g 

had  appeared  in  the  CBD  announcement. 

With  his  announcements  completed  the  contract  negoti- 
ator introduced  the  COTR  and  turned  the  briefing  over  to 
him.  In  his  presentation  the  COTR  briefly  described  the 
ETIP  program  and  the  objectives  of  the  specific  procure- 
ments. He  also  indicated  the  interrelationship  among  the 
various  items  to  be  included  in  the  solicitation  document 
(Transcript,  Note  63).  After  this,  he  introduced  the  chief 
of  the  procurment  program  for  a description  of  the  substan- 
tive area  for  which  the  systems  were  to  be  developed. 


The  announcement  had  indicated  that  the  proposed  level 
of  effort  of  the  contracts  was  100  man-months  each  rather 
than  the  actual  forecast  which  was  200  man-months  each. 

A corrected  copy  of  the  announcement  was  made  available 
in  the  briefing  room. 


The  chief  of  the  procurement  program  gave  a slide  talk. 
He  described  the  history  of  the  procurement  program  and 
the  philosophy  used  in  selecting  and  carrying  out  specific 
experiments.  He  also  described  products  and  incentives 
which  were  being  experimented  with  at  that  time  (Transcript, 
Note  64 ) . 

After  the  second  presentation  was  concluded  the  con- 
tract negotiator  requested  that  questions  be  submitted. 
Questions  were  accepted  annonyraously  in  writing  on  index 
cards  or  orally  from  participants.  The  questions  pertained 
to  the  following: 

o the  scope  of  the  work  and  the  projected  role 
of  the  successful  bidder  in  the  various  activ- 
ities related  to  ETIP  procurement  experi- 
ments, 

o coordination  between  the  two  contractors 
in  developing  the  systems, 

o potential  for  and  advisability  of  coordina- 
tion among  firms  in  bidding, 

o the  availability  of  the  names  of  contractors 
who  had  previously  participated  in  the  design 
of  ETIP  procurement  experiments, 

o the  projected  role  of  the  various  PAAs  in 
the  projects,  and 

o requested  clarifications  of  statements  made 
earlier  in  the  briefing  (Transcript,  Note 
65 ) . , 

The  meeting  was  adjourned  after  all  questions  were 
answered.  Participants  were  given  three  weeks  prior  to 
the  second  briefing  to  reflect  on  the  pre-bidders  briefing 
and  to  examine  the  technical  elements  of  the  solicitation. 


The  second  briefing  was  held  on  March  l8  and  was  atten- 
ded by  representatives  of  29  firms  ("Attendees  at  ETIP  pre- 
proposal conference,  Note  66).^  It  was  primarily  a question 
and  answer  session.  The  contract  negotiator  distributed 
copies  of  the  solicitation  document  and  indicated  that  it 
would  be  modified  to  require  20  copies  of  each  proposal 
rather  than  the  stated  15  (Transcript,  Note  67).  He  then 
opened  the  meeting  for  questions.  This  time  the  questions 
included  questions  on: 

o whether  the  requirement  for  oral  presentations 
remained , ' ^ 

o what  outputs  were  expected  from  Phase  I and 
Phase  II  of  the  projects  in  light  of  the 
small  number  of  procurement  experiments  which 
were  in  existence, 

0 how  the  efforts  of  the  two  contractors  would 
be  coordinated, 

o requirements  and  format  for  responding  to 

factors,  sub-factors,  and  illustrative  questions, 

o whether  a list  of  the  attendees  at  the  briefing 
would  be  available  to  aid  people  who  were 
looking  for  co-bidders, 

o the  expected  involvement  of  state  and  local 
procurement  organizations  in  the  experiments. 


Q 

Eighteen  of  these  had  been  at  the  February  briefing 
and  11  were  new. 

^^The  process  originally  called  for  each  bidder  to  make 
an  oral  presentation  on  their  proposal  to  the  proposal  eval 
uation  panel.  This  was  modified  to  allow  written  questions 
submitted  to  bidders  by  the  panel  through  the  contract  ne- 
gotiator . 
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o the  anticipated  transition  process  from  Phase 
I to  Phase  II,  whether  much  down  time  was 
expected,  and  about  how  the  200  man-months 
should  be  divided  between  Phase  I and  Phase 
II,  and 

o where  the  PFE  approach  had  been  used  previously 
(Transcript,  Mote  68). 

After  the  questions  and  answers  were  completed  the 
Director  of  ETIP  spoke  briefly.  He  thanked  the  represen- 
tatives of  firms  for  their  time  and  interest  and  told  them 
that  things  were  beginning  to  move  quite  rapidly  in  the 
procurement  program  and  that  many  more  projects  were  expec- 
ted to  be  in  place  soon  (Transcript,  Mote  69). 

With  the  briefings  concluded,  organizations  had  four 
weeks  to  determine  whether  to  bid  and  to  complete  prepara- 
tion of  their  proposals. 

IV . 8 Selecting  and  Involving  the  Evaluation  Panel 

A substantial  portion  of  the  time  prior  to  the  second 
briefing  was  spent  preparing  for  proposal  evaluation.  It 
was  during  this  period  that  the  author  arrived  (March  8, 

1976)  and  was  assigned  tasks  of  both  contacting  potential 
panel  members  to  solicit  their  participation  and  planning 
for  the  panel  meetings. 

An  effort  was  made  to  get  significant  participation 
on  the  panel  by  people  outside  of  ETIP.  Some  of  those  con- 
tacted were  from  PAAs  and  others  were  people  not  directly 
related  to  the  program.  Those  not  directly  related  included: 


o people  who  were  thought  to  be  experts  on 
some  of  the  specific  sub-factors  being  eval- 
uated , 

o people  who  were  familiar  with  some  of  the 
likely  bidders, 

o people  who  could  lend  credibility  to  the 
project  with  the  NBS  program  office,  and 

o people  who  were  responsible  for  managing 
other  government  evaluation  programs  who 
might  have  an  interest  in  the  PFE  technique 
as  a method  for  procuring  contractor  support 
(COTR,  Note  70). 

Nineteen  of  the  30  people  who  were  invited  to  join  the  panel 
were  from  organizations  other  than  ETIP  (compiled  from  lists. 
Note  71 ) . 

When  soliciting  participation  in  the  process,  both 
the  COTR  and  the  author  stressed  the  ability  of  a person 
to  limit  the  extent  of  their  participation.  They  were  told 
that  they  could  participate  by  evaluating  as  few  as  one 
sub-factor  if  their  time  was  severely  limited.  Eleven  of 
the  19  people  invited  from  outside  agencies  eventually  par- 
ticipated or  sent  representatives  to  participate  in  the 
evaluation. 

Members  of  the  panel  who  had  been  recruited  by  mid- 
March  were  called  together  for  a meeting  on  March  17  to: 

o review  the  process  for  proposal  evaluation, 

o determine  any  additional  information  which 
should  be  related  to  bidders  at  the  second 
bidders  briefing  on  March  18, 


o distribute  copies  of  the  the  technical  por- 
tions of  the  solicitation  document  and  a 
list  of  supporting  documents  which  had  been 
made  available  to  bidders,  and 

o review  the  schedule  for  the  April  factor 
team  meetings  (COTR,  Note  12). 

Prior  to  this  meeting  Thompson  had  written  a document  en- 
titled "Proposal  Evaluation  Procedure"  and  had  prepared 
the  forms  which  would  be  used  by  the  panel  during  the  eval- 
uation process  (Thompson,  Note  73).  One  form  was  for  rating 
the  bidders  on  sub-factors  and  factors  and  the  other  form 
was  for  use  in  obtaining  clarification  from  bidders  between 
the  time  proposals  arrive  and  the  time  that  factor  ratings 
are  assigned. 

At  the  meeting  the  documents  from  the  solicitation 

and  the  list  of  supporting  documents  which  had  been 

furnished  to  the  bidders  were  distributed  to  the 

1 1 

11  invited  panel  members  who  were  present.  Panel  members 
were  encouraged  to  at  least  scan  through  all  materials  some- 
time in  the  month  that  remained  prior  to  the  arrival  of 
proposals.  The  evaluation  process  was  then  reviewed  and 
much  of  the  remainder  of  the  meeting  was  spent  in  an  argu- 
ment concerning  the  weighting  of  sub-factors  within  the 
factors . 


1 1 

Three  of  these  later  dropped  off  of  the  evaluation 

panel . 
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At  least  two  of  the  people  present  felt  strongly  that 
the  relative  weights  of  sub-factors  within  each  factor  should 
be  pre-determined  by  the  COTR.  Their  reasons  were  that 
they  felt  that: 

0 waiting  until  individual  sub-factor  scores 
are  known  with  respect  to  the  names  of  the 
proposers  presents  an  appearance  of  conflict 
of  interest,  with  the  factor  team  being  able 
to  manipulate  the  weights  in  favor  of  a par- 
ticular proposal  ("Memorandum,”  Note  7^; 
"Memorandum,"  Note  75),  and 

o a great  amount  of  time  would  be  taken  trying 
to  reach  agreement  on  the  weights  in  the 
team  meetings  ("Memorandum,"  Note  76). 

The  argument  was  quite  heated  and  resulted  in  several  of 

the  participants  writing  memos  afterwards  to  clarify  their 

positions  ("Memorandum,"  Note  77;  "Memorandum,"  Note  78; 

COTR,  Note  79).''^ 

The  COTR  decided  not  to  request  a change  in  the  process. 
He  felt  that  one  substantial  input  which  the  panel  members 
could  have  would  be  assessing  the  relative  importance  of 
sub-factors.  He  also  felt  that  the  time  for  decisions  would 
not  be  adversely  affected  by  the  lack  of  pre-assigned  weights 
(COTR,  Note  80). 

Another  change  in  the  process  was  proposed  at  the  meet- 
ing by  the  representative  of  the  NBS  program  office.  He 
suggested  that  an  executive  committee  be  established  to 


1 2 

Two  of  these  people  were  among  the  three  at  the  meet- 
ing who  eventually  dropped  off  of  the  panel. 
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review  the  decisions  made  by  the  panel.  The  COTR  rejected 
this  proposal  after  conferring  with  the  contract  negotiator. 
The  contract  negotiator  advised  against  such  a committee. 

He  felt  that  the  process  would  provide  him  with  a clear 
indication  of  the  technical  merits  of  the  proposals  and 
did  not  desire  a panel  which  would  filter  these  recommenda- 
tions (COTR,  Note  8l).  In  his  follow-up  memo  to  those  who 
attended  the  meeting,  the  COTR  expressed  his  faith  in  the 
panel  and  the  process  and  said  that  he  felt  that  such  a 
review  committee  would  be  unlikely  to  have  sufficient  famil- 
iarity with  the  individual  proposals  to  be  effective  and 
would  give  the  appearance  of  second  guessing  the  panel  (COTR, 
Note  82 ) . 

This  was  the  only  time  the  panel  met  as  a group  prior 
to  the  arrival  of  proposals.  The  remainder  of  the  time 
prior  to  the  closing  date  was  spent  recruiting  additional 
panel  members  and  making  assignments  of  individuals  to  spe- 
cific sub-factors.  In  general  assignments  were  made  so 
that  individuals  would  evaluate  sub-factors  where  they  had 
a special  expertise  or  interest.  In  several  cases,  where 
individuals  were  expected  to  have  a close  relationship  with 
the  projects,  they  were  assigned  to  a wide  range  of  sub- 
factors. Each  sub-factor  had  at  least  two  people  assigned 


to  it . 


IV. 9 The  Proposals  Arrive 


The  closing  date  for  receipt  of  proposals  was  April 
15.  On  this  day  the  contracting  office  received  eight  pro- 
posals. Five  were  for  consideration  as  the,  agency  impact 
contractor  and  two  were  for  consideration  as  the  commercial 
impact  contractor.  ^ One  firm  asked  that  its  proposal  be 
considered  for  both.  Another  firm  had  submitted  separate 
proposals  for  each  contract.  That  is,  seven  firms  bid. 

Four  bid  on  only  the  agency  impact  contract,  one  bid  on 
only  the  commercial  impact  contract  and  two  bid  on  both 
contracts . 

The  proposal  evaluation  panel  was  convened  the  morning 
of  April  16 . Nineteen  of  the  23  panel  members  were  able 
to  participate.  The  first  agenda  item  to  be-  covered  was 
to  review  the  evaluation  process.  Members  were  asked  to 
look  at  the  "Proposal  Evaluation  Process"  document  in  their 
packet  as  it  was  reviewed  with  them.  They  were  reminded 
that  there  were  two  minimum  requirements  on  each  member 
of  the  evaluation  team.  These  were: 
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The  proposal  submitted  by  one  firm  was  not  clear  as 
to  which  contract  the  firm  wished  it  be  evaluated  for. 

The  contract  negotiator  was  called  and  requested  to  obtain  , 
a clarification  from  the  firm.  The  firm  indicated  that 
it  was  submitting  for  only  the  commercial  impact  contract. 
The  factor  teams  on  agency  impact  included  this  firm  in 
their  evaluation  until  the  clarification  was  obtained. 

These  ratings  are  not  included  in  the  data  analysis  in  Chap- 
ter V. 


1 ) If  a panel  member  finds  ambiguity  or  missing 
information  which  will  substantially  inter- 
fere with  their  ability  to  rate  a bidder, 

it  needs  to  be  uncovered  right  away  so  the 
contract  negotiator  can  obtain  an  answer 
back  from  the  bidders  in  time  for  evaluation 
(question  forms  were  provided  to  panel  mem- 
bers for  this  purpose). 

2)  Panel  members  assigned  to  a sub-factor  need 
to  fill  oub  one  of  the  rating  forms.  The 
names  of  the  bidders  are  to  be  put  in  the 
blank  space.  The  rating  can  be  based  solely 
upon  reading  that  one  section  of  the  pro- 
posal. What  they  want  to  read  beyond  that 
is  up  to  each  individual  (Tape  of  meeting, 

Note  83 ) . 

Panel  members  were  cautioned  that  they  were  not  to  acquire 
additional  information  from  bidders  except  through  the  con- 
tract negotiator  (Tape  of  meeting.  Note  84). 

Members  were  reminded  that  the  factor  team  was  respon- 
sible for  coming  up  with  a single  rating  across  the  factor 
for  every  one  of  the  bidders  and  that  each  team  had  control 
of  most  of  the  process  by  which  they  chose  to  do  this. 

The  ground  rules  were: 

o each  individual  was  required  to  submit  a 
signed  sub-factor  rating  sheet  at  the  start 
of  the  session, 

o the  team  had  to  reach  an  agreement  on  the 
ratings  without  voting, 

o all  members  of  the  team  were  to  sign  the 

factor  rating  form  (any  comments,  references, 
and  reservations  were  also  to  be  included 
as  a part  of  the  record),  and 

o the  sessions  were  to  be  taped  (Tape  of  meet- 
ing, Note  85 ) . 

The  method  for  reaching  agreement,  including  the  order  of 
consideration  of  sub-factors,  the  relative  weighting  of 


the  sub-factors  in  the  final  factor  weightings,  and  whether 
to  consider  the  factor  as  a whole  rather  than  rating  the 
sub-factors  were  up  to  each  team  (Tape  of  meeting,  Note 
86). 

The  panel  was  told  that  results  of  the  individual  fac- 
tor teams  would  be  combined  using  the  factor  weights  listed 
in  the  RFP.  A narrative  description  of  the  results  of  the 
panel  process  would  be  prepared  for  the  contract  negotiator. 
This  would  be  followed  by  meetings  for  clarification  with 
the  contending  bidders  (Tape  of  meeting.  Note  87). 

Individuals'  assignments  to  sub-factors  were  reviewed, 
factor/sub-factor  rating  sheets  were  distributed,  copies 
of  the  proposals  were  signed  out  to  panel  members  and  the 
meeting  broke  for  lunch  (Tape  of  meeting.  Note  88).  During 
the  break  a tentative  schedule  of  factor  team  meetings  was 
worked  up  by  charting  the  schedule  conflicts  which  panel 
members  had  during  the  next  week.  The  best  schedule  which 
could  be  worked  out  resulted  in  several  panel  members  being 
unable  to  participate  due  to  schedule  conflicts. 

The  remainder  of  the  day  on  April  16 , was  allocated 
to  initial  review  of  the  proposals.  Two  question  forms 
were  submitted  which  pointed  out  some  sections  which  were 
apparently  missing  from  proposals  submitted  by  two  different 
bidders  (Question  for  bidder  #101,  Note  89;  Question  for 
bidder  #102,  Note  90).  These  questions  were  called  in  to 


the  contract  negotiator  for  transmittal  to  the  bidders. 
Figure  IV. 4 is  a copy  of  the  form  which  was  used. 

Team  members  who  were  unable  to  attend  the  April  16, 
meeting  were  briefed  individually  and  provided  with  copies 
of  the  proposals  and  the  schedule. 

IV. 10  The  Factor  Teams  Meet 

IV. 10.1  Introduction 

For  the  two  projects  there  were  a total  of  fourteen 
factors  and  thus  fourteen  teams.  All  fourteen  factor  team 
meetings  were  held  during  the  week  of  April  19th  (see  Figure 
IV. 5).  They  were  scheduled  at  a rate  of  three  factors  per 
day  with  the  exception  of  the  first  day,  for  which  only 
two  meetings  were  scheduled.  The  nominal  time  allocated 
for  each  meeting  was  two  hours. 

1 4 

The  meetings  were  held  in  ETIP’s  conference  room. 

The  room  was  equipped  with  a blackboard,  a tape  recorder 
and  extra  rating  and  question  forms. 

Each  meeting  having  new  participants  who  had  not  been 
in  earlier  meetings  was  begun  with  a brief  discussion  of 
the  evaluation  procedures  and  a decision  on  how  to  proceed. 
The  meetings  then  continued  until  scores  were  determined 
for  all  bidders  on  each  sub-factor  and  on  the  factor  as 
a whole. 


An  exception  was  the  first  factor  team  meeting  which 
was  held  in  a larger  NBS  lecture  room. 
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RFP  6-35756 


' C3UESTION  FOPM 

a □ Requested  by: 

Bidder:  , Date: 


Factor  (or  svib- factor) : 


Proposed 

Specific 

Question: 


Qlto  the  bidder  Note:  Write  the  qxiestion  as 

Q lb  (specify) : specifically  as  possible  in  a form 

that  the  bidder  can  respond  to 
directly.  Give  this  form  to  the  CCTR. 


Question  (Revised,  if  necessary) : Date  given  to  CO: 

Date  answer  received: 


FIGURE  IV. 4 THE  QUESTION  FORM  FOR  USE  IN  OBTAINING 

CLARIFICATIONS 
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FIGURE  IV. 5 SCHEDULE  OF  FACTOR  TEAM  MEETINGS 


What  follows  in  this  section  are  examples  of  how  the 
team  meetings  worked.  Anecdotes  are  presented  from  the 
meetings  to  illustrate  the  types  of  issues  which  arose  and 
how  they  were  resolved.  They  are  presented  to  illustrate 
the  types  of  discussions  which  took  place  over  the  course 
of  the  evaluation.  They  are  not  meant  to  represent  the 
total  discussion  during  any  of  the  meetings.  The  sources 
of  these  are  the  tape  recordings  which  were  made  of  each 
of  the  meetings.  Displays  of  the  initial  scores  for  each 
rater  and  the  final  scores  for  the  teams  are  included  in 
the  appendix  for  readers  who  wish  to  follow  the  actual  num- 
bers from  the  evaluation  process.  The  names  of  individuals 
involved  in  the  discussion  and  firms  being  rated  are  not 
included  in  these  descriptions  or  in  the  appendix. 

The  first  factor  scheduled  to  be  evaluated  was  Factor 
VI,  Data  Collection  Processes  on  the  agency  impact  contract. 
This  factor  was  selected  as  the  first  because  it  was  felt 
to  be  less  difficult  to  evaluate  than  some  of  the  others, 
e.g.,  management,  and  would  be  a good  one  for  panel  members 
to  get  a feeling  for  how  the  process  functioned.  All  team 
members  had  been  encouraged  to  come  and  observe  the  first 
meeting . 


Ill 


IV. 10. 2 The  First  Factor  Team  Meeting 
The  first  factor  team  meeting,  Agency  Impact,  Factor 
VI,  Data  Collection  Process,  began  Monday,  April  19,  1976, 
at  approximately  10  A.M.  The  four  assigned  panel  members 
were  all  present.  The  tape  recorder  was  turned  on  and  the 
meeting  began  with  some  discussion  on  how  to  proceed.  Ground 
rules  were • quickly  reviewed  and  the  first  set  of  signed 
sub-factor  rating  forms  were  handed  in  by  panel  members. 
(Figure  IV. 6 is  an  example  of  what  a completed  subfactor 

rating  form  looked  like.)  Some  time  was  taken  getting  the 

« 

scores  posted  on  the  blackboard  in  order  to  determine  a 
method  which  would  make  the  distributions  clear  (Figure 
IV. 7 illustrates  the  form  that  the  postings  took.)  This 
was  followed  by  discussion  on  how  to  proceed  in  order  to 

k 

obtain  scores  for  each  bidder.  One  panel  member  suggested 
averaging  the  scores.  Another  argued  that  this  would  not 
be  acceptable  since  there  was  a need  to  know  why  one  per- 
son’s score  for  a bidder  was  a nine  while  another  person 
scored  the  same  bidder  at  one. 

Discussion  on  the  individual  proposals  began  with  peo- 
ple giving  the  reasons  behind  their  ratings.  A panel  member 
inquired  about  the  extent  to  which  personal  knowledge  about 
bidders  was  admissible  and  he  was  assured  by  another  member 
that  it  was  a very  proper  thing  to  put  onto  the  table. 

Another  question  was  whether  the  panel  should  be  concerned 
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FACTOR  (AND  SUB-FACTOR)  WORKSHEET 


Factor  or  sub- factor  (number  & title) ; 


VI.]  Instrument  Design 


When  used  by  individual  team  menbers  for 
siib-f actor  ratings: 

Signature : EVAL  205 

Date:  April  ] 9, 1976 


When  used  by  team  to  record  final 
factor  ratings  only: 

Signatures: 


Date: 


Nothing  can  stop 
him  from  being 
a success 

Should  perform 
well 


ID 


B205 


Almost  as  good 


Acceptable  as  is 


B206 


Acceptable  ’.»ath 
negotiation 


B203  B204 


B202 

Cll  Qiedc  if  continuation  sheet  acoorpanies  this  form 


Marginal  at  best 


Unacceptable 
, without  significant 
• revisions 

i Possibly  still 
I xmacceptable  with 
' revisions 

Probably  still 
, unacceptable  with 
revisiOTis 

No  hope  (totally 
unacceptable) 


Note:  Individual  team  members  may  use  form  as  worksheet,  making  dianges,  erasures, 

etc.,  and  adding,  notes.  Prior  to  the  beam  meeting,  ratings  should  be  entered 
in  ink.  Subsequent  notes  and  revisions  may  be  made  during  the  meeting  by 
cross-out,  etc. , and  the  resultant  form  will  become  part  of  the  record. 


FIGURE  IV. 6 A COMPLETED  SUB-FACTOR  RATING  FORM 
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at  this  point  about  whether  certain  people  in  a winning 
bidder's  organization  would  be  tied  to  the  contract  through 
a key  personnel  clause.  He  was  told  that  anything  which 
the  factor  team  was  concerned  about  which  influenced  its 
rating  should  be  noted  in  the  record  of  the  process  so  that 
it  could  be  discussed  with  potential  winners. 

It  was  pointed  out  during  this  and  other  panel  meetings 
that  all  bidders  falling  into  the  competitive  range  would 
be  called  in  for  discussion  and  clarification  prior  to  final 
evaluation.  It  was  therefore  important  to  enter  notes  into 
the  record  where  the  panel  had  uncertainty  about  their  rat- 
ings. The  related  issues  would  then  be  raised  with  bidders 
during  the  subsequent  discussions  in  order  to  reduce  the 
range  of  uncertainty. 

The  key  point  of  disagreement  on  sub-factor  VI. 1 was 
whether'a  specific  bidder  did,  in  fact,  have  the  in-house 
capability  to  perform  in  the  area  of  instrument  design. 

The  issue  was  resolved  by  agreeing  that  the  bidder  was  prob- 
ably acceptable  on  the  sub-factor  but  the  remaining  uncer- 
tainty would  be  recorded  with  a note  in  the  record.  The 
note  was  to  be  sure  that  they  have  at  least  one  person  inte- 
grated into  their  team  who  is  strong  on  instrument  design 
in  the  event  that  they  are  found  to  be  strong  enough  overall 
to  be  considered  competitive. 

As  the  panel  arrived  at  a score  for  each  of  the  bid- 
ders, that  bidder's  score  was  posted  on  the  board  under 
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the  sub-factor  column  and  removed  from  under  the  individual 
rater  columns  (see  Figure  IV. 8).  After  all  bidders  had 
been  rated  on  a sub-factor  the  left  side  of  the  board  was 
clean  and  the  column  for  the  sub-factor  score  was  complete 
(Figure  IV. 9).  Panel  members  scores  were  then  posted  under 
their  column  for  the  next  sub-factor  (Figure  IV. 10)  and 
the  same  process  was  repeated. 

There  was  a great  deal  of  variability  in  the  amount 
of  discussion  required  under  each  sub-factor  to  arrive  at 
scores  for  different  bidders.  As  would  be  expected,  it 
seemed  that  in  cases  where  there  was  substantial  initial 
agreement  the  time  would  be  brief.  Conversely,  the  time 
required  would  generally  be  longer  in  cases  where  at  least 
one  panel  member’s  initial  rating  varied  from  those  of  others 
on  the  panel.  Often  the  panel  began  discussion  on  bidders 
for  which  there  was  greater  initial  agreement.  This  enabled 
the  panel  to  make  progress  in  rating  and  at  the  same  time 
discuss  some  of  the  issues  related  to  the  sub-factor  without 
strong  arguments  over  the  bidders. 

Several  times  during  the  first  meeting  there  was  discus- 
sion about  the  significance  of  the  different  numbers  on 
the  rating  form.  This  discussion  seemed  to  establish  the 
interpretation  of  the  numbers  for  the  remaining  factor  team 
meetings.  The  distinction  between  a score  of  six  and  a 
score  of  seven  was  emphasized.  Using  the  words  next  to 
the  numbers  on  the  form  as  a guide,  it  was  felt  that  any 
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FIGURE  IV. 8 ASSIGNMENT  OF  THE  FIRST  SCORE  ON  SUB-FACTOR  VI. 
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bidder  who  was  rated  at  seven  or  above  was  thought  to  have 
an  acceptable  probability  of  performing  with  respect  to 
the  issues  raised  by  the  sub-factor.  A bidder  was  rated 
at  a six  or  below  if  there  was  some  doubt  that  the  bidder’s 
performance  would  be  acceptable.  This  meant  that  in  the 
event  that  the  bidder  appeared  to  be  acceptable  overall 
they  would  be  called  in  for  discussions.  These  discussions 
would  be  used  to  resolve  existing  uncertainties  over  the 
bidders’  scores,  i.e.,  to  clarify  whether  the  bidder  could 
be  expected  to  perform  well  on  the  sub-factor  if  they  were 
under  contract.  A six  reflected  a modest  doubt  as  to  the 
bidder’s  capability.  The  bidder  was  rated  at  a five  if 
the  panel  had  more  serious  doubts  that  the  bidder  would 
perform  acceptably.  Scores  of  four  or  below  were  used  if 
the  panel  had  very  serious  doubts  about  the  bidder’s  present 
or  potential  capability  with  respect  to  the  sub-factor  is- 
sues. A bidder  with  more  than  one  or  two  fours  on  sub- 
factors was  thought  to  be  unacceptable  unless  they  were 
rated  significantly  higher  elsewhere  and  could  demonstrate 
that  they  could  improve  their  score  on  these  low  sub- 
factors . 

In  arriving  at  sub-factor  scores  the  panel  was 
often  reminded  by  its  members  that  it  was  to  assess 
bidders  using  the  criteria  of  the  bidder’s  understanding 
of  the  sub-factor,  the  proposed  approach  for  dealing 
with  the  issues  raised  by  the  sub-factors,  and  the 


bidder’s  present  or  potential  capability  for  dealing  with 
the  issues.  There  were  cases  in  the  first  factor  team  meet- 
ing, and  in  many  of  the  others,  where  the  panel  felt  that 
the  bidder’s  response  indicated  an  acceptable  understanding 
and  approach  but  there  was  a reasonable  doubt  that  they 
had  the  capability  to  perform. 

As  an  example  there  was  one  bidder  who  had  a team  that 
was  quite  heavily  loaded  with  consultants.  It  appeared 
that  the  understanding  of  the  issues  was  held  by  these  con- 
sultants but  it  did  not  appear  that  these  consultants  were 
integrated  effectively  into  the  decision  making  process 
for  the  project.  In  this  case  the  bidder  was  given  a score 
of  five,  with  the  understanding  that  this  point  would  be 
investigated  during  discussions  if  the  bidder  were  in  the 
competitive  range. 

Ratings  seemed  to  be  made  on  both  an  absolute 
and  a relative  basis.  Bidders’  responses  were  dis- 
cussed as  they  compared  with  the  standards  which  each  panel 
member  used  in  applying  the  three  evaluation  criteria  to 
their  ratings.  There  was  also  discussion  across  bidders 
of  the  relative  degree  to  which  they  dealt  with  the  sub- 
factor issues.  For  example,  a panel  member  would  indicate 
the  evidence  he  used  in  his  rating  of  a specific  bidder 
and  then  contrast  it  with  what  another  bidder  said  as  an 
indication  of  why  he  rated  that  bidder  either  the  same  or 
differently . 


After  all  of  the  sub-factors  had  been  rated  in  factor 


VI,  the  panel  was  prepared  to  determine  the  factor  score 
for  each  bidder.  There  was  some  discussion  of  how  the  var- 
ious sub-factors  should  be  weighted.  The  statement  of  the 
factor  was  read  and  the  panel  members  proceeded  to  assign 
the  factor  score  by  visually  averaging  the  sub-factor  scores. 
No  one  had  a compelling  desire  to  weight  any  sub-factor 
greater  than  any  other.  The  scores  were  posted  on  the  board 
(Figure  IV. 11)  and  when  the  process  was  complete  they  were 
copied  onto  a factor  rating  sheet.  The  sheet  was  labeled 
with  the  factor  name,  dated,  and  passed  to  each  member  of 
the  panel  for  a signature.  (Figure  IV. 12  is  an  example 
of  what  a completed  factor  rating  form  looked  like.)  Deter- 
mining factor  scores  once  the  sub-factor  scores  were  com- 
plete took  about  five  minutes.  The  first  panel  meeting 
was  concluded  two  hours  after  it  began. 

IV. 10. 3 The  Second  Factor  Team  Meeting 
The  second  meeting  was  Agency  Impact,  Factor  V,  Evalua- 
tion Design.  It  began  Monday,  April  19,  at  about  3 P.M. 
with  the  seven  panel  members  present.  After  a brief  review 
of  ground  rules,  the  ratings  for  the  first  sub-factor  were 
posted  and  discussion  began. 

During  rating  on  the  first  sub-factor  (V.l)  there  was 
a case  where  information  presented  by  a bidder  as  a strong 
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ASSIGNMENT  Or  FACTOR  SCORES  FOR  FACTOR  VI 
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FfiCTOR  (AND  SUB-FA3TDR)  WORKSHEET 
Factor  or  sub-factor  (number  & title) : 


VI  Data  Collection  Process 


When  used  by  team  to  record  final 
factor  ratings  only: 


When  used  by  individual  team  msnbers  for 

EVAL208 

sub-factor  ratings: 

EVAL210 

1 

Signature: 

EVAL211 

Date:  Date 

: April  19,  1976 

— 

1 

! 

Nothing  can  stop 
him  from  being 

10 

I 

a success 

Should  perform 

1 

* 

well 

9 

1 

‘ B205  1 

Almost  as  good 

8 

B203 

1 

Acceptable  as  is 

7 

1 - 

Acceptable  with 

6 

B202  ; 

1 

negotiation 

I 

B206 

1 

Marginal  at  best 

5 

' 

tfriacceptable 

B201  B204  , 

without  significant 

4 

revisions 

1 

1 

Possibly  still 
unacceptable  with 

5 

Note:  Should  B202  get  the  contract  they' 
must  bring  someone  on  board  who  is 

revisions 

competent  in  instrument  design  (VI. 1).  • 

Probably  still 

B202's  performance  must  be  further 

unacceptable  with 

2 

examined  on  sub-factor  VI. 3. 

revisions 

■ 

No  hope  (totally 

Check  if  ocntinuation  sheet  accompanies  this  form  i 

unacceptable) 

i 

Note:  Individual  team  ment)ers  may  use  form  as  worksheet,  making  changes,  erasures, 

etc.,  and  adding  notes.  Prior  to  the  team  meeting,  ratings  should  be  entered 
in  ink.  Subsequent  notes  and  revisions  may  be  made  during  the  meeting  by 
cross-out,  etc. , and  the  resultant  form  will  beccne  part  of  the  record. 


FIGURE  IV. 12 


A COMPLETED  FACTOR  RATING  FORM 


example  of  their  capability  in  the  area  of  the  sub-factor 
was  disputed  by  a panel  member  with  outside  information. 

The  bidder  had  claimed  to  be  making  a major  contribution 
in  a substantive  area  in  which  the  panel  member  claimed 
to  be  knowledgeable.  The  panel  member  indicated  that  he 
had  just  completed  a major  review  of  work  in  this  area  and 
was  unaware  of  this  bidder's  involvement  in  the  field. 

It  is  unclear  how  much  this  information  affected  the  final 
score  for  this  bidder  since  others  had  argued  in  the  firm’s 
favor.  Panel  members  had  rated  the  firm  between  four  and 
seven  and  a half  and  after  discussion  the  team,  settled  on 
a f i ve . 

When  rating  another  bidder  two  panel  members  disagreed 
about  prior  performance  of  members  of  the  bidder’s  proposed 
team  in  the  area  of  evaluation  design.  This  resulted  in 
a discussion  of  the  characteristics  of  the  prior  work  in 
question  and  its  relationship  to  the  present  work  and  the 
standards  of  judgement  that  each  of  these  two  panel  members 
was  using.  The  rating  was  resolved  by  the  panel  in  favor 
of  the  claim  that  the  work  which  the  bidder  had  done  effec- 
tively in  the  past  was  not  sufficiently  comparable  to  what 
was  required  here. 

In  discussion  on  a rating  for  another  bidder,  a panel 
member  from  outside  of  ETIP  was  familiar  with  their  prior 
performance.  There  had  been  some  argument  for  a low  score 


and  this  panel  member  was  able  to  cite  prior  positive  exper- 
ience with  the  people  from  this  bidder’s  organization  to 
influence  a higher  rating  for  them  than  was  desired  ini- 
tially by  some  of  the  ETIP  raters. 

The  rating  of  sub-factor  V.1  presented  the  panel  with 
its  first  difficult  problem  in  achieving  consensus  on  a 
single  rating  for  a bidder.  After  panel  members  had  pre- 
sented their  views  on  the  bidder  there  remained  a strong 
disagreement  between  two  panel  m.embers.  One  would  not  rate 

them  below  a six  and  the  other  would  not  come  above  a four. 

$ 

The  panel  agreed  to  leave  this  temporarily  unresolved. 

The  bidder  was  put  at  both  four  and  six  on  the  board  with 
brackets  around  both  scores  indicating  that  the  score  would 
be  resolved  after  the  other  sub-factors  had  been  discussed 
and  before  assigning  the  factor  score.  Eventually  they 
were  given  a score  of  six  on  this  sub-factor  and  six  on 
the  factor.  This  technique  was  used  several  times  during 
the  week  and  proved  to  be  useful  in  keeping  the  evaluation 
process  moving  and  in  resolving  differences. 

The  time  to  accomplish  the  rating  for  the  first  sub- 
factor was  about  75  minutes.  Lengthy  discussion  on  the 
first  sub-factor  tended  to  be  a pattern  throughout  the  fac- 
tor meetings.  The  arguments  which  were  made  provided  a 
context  for  the  discussion  on  other  sub-factors  which  then 
proceeded  more  rapidly  (Figure  IV. 13).  The  ratings  on  sub- 
factor V.2  were  accomplished  in  about  five  minutes. 
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This  factor  team  meeting  (factor  V)  lasted  a 
little  over  two  and  a half  hours.  Since  it  was  held 
at  the  end  of  the  day,  three  of  the  team  members 
had  to  leave  after  the  fourth  sub-factor,  when  the 
time  got  late. 

In  this  case,  and  in  several  others  during  the  week, 
panel  members  were  unable  to  be  present  for  part  or  all 
of  the  team  meeting.  If  they  had  furnished  their  signed 
rating  forms  to  the  panel,  their  ratings  were  posted  on 
the  blackboard  as  a reference  point  for  the  team.  If  the 
team  felt  it  needed  to  inquire  into  the  reason  for  a rating 
of  a person  who  was  not  present,  that  person  could  be  called. 
This  option  was  used  in  one  of  the  later  meetings  where 
the  absent  team  member  had  rated  a bidder  higher  than  other 
team  members  who  were  present. 

IV. 10. 4 The  Third  Factor  Team  Meeting 

The  third  factor  team  met  at  approximately  9 A.M. 
on  Tuesday,  April  20.  This  was  factor  III,  Objectives  for 
Agency  Impact.  The  team  had  six  members,  one  who  arrived 
after  discussion  of  the  first  sub-factor  had  been  completed. 

One  issue  which  was  pointed  out  during  this  meeting 
was  that  at  least  two  of  the  team  members  were  tending  to 
take  extreme  positions  when  rating  certain  bidders.  This 
was  because  one  member  stressed  the  importance  of  bidders’ 
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qualifications  with  respect  to  the  substantive  area  of  pro- 
curement, while  the  other  member  stressed  the  importance 
of  qualifications  in  methodology.  They  tended  to  rate  bid- 
ders as  extremely  high  or  extremely  low  on  many  sub- factors 
according  to  these  biases.  In  effect  they  were  evaluating 
the  proposal  as  a whole  on  each  sub-factor.  This  led  to 
discussion  on  the  importance  of  various  experiences  and 
skills  and  how  they  relate  to  potential  performance  with 
respect  to  the  specific  sub-factor  issues.  Discussion  and 
recommendation  of  consensus  ratings  by  other  team  members 
was  important  in  forcing  team  members  to  examine  biases 
against  the  specific  requirements  of  each  sub-factor. 

IV. 10. 5 The  Fourth  Factor  Team  Meeting 

The  fourth  team  to  meet  was  Factor  I,  Management  for 
Agency  Impact.  This  team  met  on  Tuesday,  April  20,  at  12:30, 
with  four  team  members. 

Much  of  the  discussion  during  this  meeting  was  about 
the  distinction  between  capability  to  do  general  project 
management  and  the  capability  to  manage  this  specific  pro- 
ject to  develop  evaluation  systems.  At  least  one  panel 
member  felt  that  the  former  was  sufficient.  Others  argued 
the  merits  of  experience  in  the  management  of  specific  re- 
lated types  of  work. 

There  was  also  discussion  on  the  commitment  of  the 
firms  to  provide  a full  time  team  for  the  necessary  two 
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to  three  years  of  the  project.  Sub-factor  II. 2 specifically 
raised  the  issue  of  the  internal  commitment  of  the  firm 
to  the  project.  On  this  sub-factor  the  team  was  unable 
to  differentiate  among  bidders  from  any  readily  available 
information.  It  was  decided  that  all  bidders  would  be  rated 
at  6,  and  the  necessary  clarification  on  this  sub-factor 
would  be  obtained  during  any  subsequent  oral  discussions 
with  the  bidders. 

IV. 10. 6 The  Fifth  Factor  Team  Meeting 

Agency  Impact,  Factor  IV  met  on  Tuesday,  April  20, 

# 

at  3 p.m.  Five  panel  members  were  present  for  the  meeting. 

In  this  meeting  there  was  a situation  in  which  one 
panel  member  was  arguing  for  a rating  of  four  for  a bidder 
on  a sub-factor.  Two  other  panel  members  had  rated  the 
bidder  at  seven  and  one  had  rated  them  at  nine.  The  high 
rating  was  from  a subordinate  of  the  person  who  gave  the 
low  rating. 

One  of  the  people  who  rated  the  bidder  at  seven  indi- 
cated that  he  had  used  information  which  the  bidder  provided 
under  another  sub-factor  (within  the  factor)  to  support 
his  relatively  higher  rating  on  this  one.  The  person  who 
was  at  a nine  indicated  that  he  would  be  willing  to  come 
down  to  a seven  if  his  boss  would  come  up.  At  this  point, 
the  low  rater  indicated  that  he  would  not  want  to  come  above 


a six . 


There  was  additional  discussion  in  which  the  panel 
member  who  had  found  information  elsewhere  in  the  proposal 
pointed  out  the  specific  sentences  which  he  felt  were  rel- 
evant. The  low  panel  member  eventually  agreed  to  com.e  up 
to  seven,  indicating,  however,  that  he  was  uncomfortable 
in  doing  so. 

IV. 10. 7 The  Sixth  Factor  Team  Meeting 

Commercial  Impact,  Factor  III,  Objectives,  met  on  Vvednes- 
day,  April  21,  at  10  A.M.  Five  panel  members  were  present. 

In  determining  ratings  under  this  factor  there  was  a substan- 
tial amount  of  discussion  concerning  the  appropriateness 
of  various  sub -objectives  as  related  to  the  overall  objec- 
tive of  developing  the  systems.  One  panel  member  explained 
that  he  had  rated  a bidder  low  on  a sub-factor  where  the 
bidder  had  indicated  that  ETIP  should  do  some  simplistic 
evaluations  early  in  the  project  in  order  to  have  some  favor- 
able outcomes.  He  felt  that  ETIP's  objective  was  to  produce 
credible  results.  Another  panel  member  argued  for  the  bid- 
der’s position  and  stressed  the  importance  of  ETIP  having 
some  successes  in  the  first  year  in  order  to  build  constit- 
uency support  for  the  evaluation  systems.  The  first  panel 
member  was  persuaded  and  raised  his  rating. 

Another  series  of  discussions  on  this  factor  surrounded 
the  match  between  the  approach  proposed  by  a bidder  and 
the  objectives  of  the  project.  The  bidder  had  been  develop- 
ing a specific  model  which  they  believed  was  applicable 
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to  ETIP’s  requirements.  The  proposal  indicated  that  they 
would  further  test  and  develop  this  model  using  the  various 
procurement  experiments.  One  panel  member  stressed  that 
this  bidder  wished  to  pursue  his  own  independent  objective 
of  developing  his  model  and  was  not  considering  other  meth- 
ods which  might  be  far  more  appropriate  under  various  circum- 
stances which  would  occur  during  the  project.  This  panel 
member  was  successful  in  persuading  others  on  the  team  to 
reduce  their  ratings. 

IV. 10. 8 The  Seventh  Factor  Team  Meeting 

* 

The  next  meeting  was  Agency  Impact,  Factor  VII,  Data 
Analysis  Process.  It  met  Wednesday,  April  21,  at  12:30 
with  four  panel  members.  During  this  factor  meeting  a discus 
sion  occurred  concerning  the  capability  of  a bidder  who  ^ 
admitted  to  a lack  of  capability  in  dealing  with  a priori 
proposition  testing.  One  team  member  used  this  as  a justi- 
fication for  his  low  rating.  He  was  reminded  by  another 
team  member  that  the  standard  for  evaluation  is  present 
or  potential  capability.  Since  the  bidder  seemed  to  have 
a good  understanding  of  where  their  specific  weakness  was 
it  was  argued  that  they  had  a strong  potential  capability. 

The  panel  settled  on  a score  of  seven. 

This  was  one  factor  where  the  team  made  specific  use 
of  a panel  member  from  outside  of  ETIP  who  was  an  expert 
in  the  substantive  area  of  the  factor.  This  panel  member 


presented  discussion  on  what  knowledge  and  skills  he  felt 
contractors  would  need  to  have  in  order  to  match  with  what 
he  saw  as  the  requirements  of  the  project.  He  also  discus- 
sed the  appropriateness  of  the  data  analysis  techniques 
which  various  bidders  were  suggesting. 

During  this  meeting  one  team  member  inquired  as  to 
whether  it  was  possible  to  determine  which  bidders  could 
no  longer  be  considered  acceptable  and  to  save  time  by  elim 
inating  these  bidders  from  further  rankings.  That  is,  sine 
this  was  the  sixth  factor  team  meeting  on  the  agency  impact 
contract  it  would  be  possible  to  determine  whether  any  bid- 
der’s ratings  were  so  low  overall  that  they  would  be  unac- 
ceptable no  matter  how  well  they  did  on  this  and  the  remain 
ing  factor  ratings.  Another  panel  member  pointed  out  that 
the  rules  of  the  procurement  did  not  allow  the  panel  to 
do  this.  The  procedure  required  evaluation  of  each  bidder 
on  all  sub-factors. 

IV. 10.9  The  Eighth  Factor  Team.  Meeting 

Agency  Impact,  Factor  II,  Reports  and  Reviews  was  the 
final  Agency  Impact  factor  team  to  meet.  The  meeting  was 
convened  at  3 P.M.  on  Wednesday,  April  21,  with  five  panel 
members.  During  this  factor  team  meeting  a lengthy  discus- 
sion took  place  when  one  panel  member  wanted  to  rate  one 
of  the  bidders  as  a 10  on  the  first  sub-factor.  The  panel 
member  claimed  that  this  was  one  of  three  sub-factors  in 
the  procurement  where  he  was  an  expert  and  that  he  felt 


that  this  particular  bidder  really  understood  the  issues 
here.  To  support  his  argument  he  referred  to  his  prior 
experience  with  other  contractors  on  other  projects  in  deal- 
ing with  these  issues.  In  the  end  the  panel  could  not  be 
convinced  to  come  above  an  eight. 

IV. 10. 10  The  Ninth  Factor  Team  Meeting 

The  Commercial  Impact,  Factor  VII,  Data  Analysis  meet- 
ing began  at  10  A.M.  on  April  22.  Four  panel  members  were 
present.  The  ground  rules  were  reviewed  since  a panel  mem- 
ber who  had  not  previously  participated  was  present.  Follow 
ing  this  brief  review  the  team  was  asked  to  turn  in  their 
first  set  of  ratings.  The  new  panel  member  indicated  that 
he  was  prepared  to  discuss  the  proposals  as  a whole  but 
that  he  had  not  attempted  to  follow  the  rating  sheet  format. 
He  was  told  that  the  rules  of  the  procurement  specified 
that  each  panel  member  is  required  to  hand  in  at  least  one 
sub-factor  rating  form  in  order  to  participate  as  a team 
member.  He  indicated  that  he  was  unwilling  to  do  that  since 
he  felt  that  rating  sub-factors  was  evalua-ting  the  issues 
out  of  context.  Discussion  with  the  other  panel  members 
continued  for  several  minutes  concerning  the  bases  for  the 
evaluation  and  what  the  new  panel  member  felt  they  should 
be.  He  indicated  that  he  would  summarize  his  views  of  the 


and  said  that  he  would 


three  proposals  in  a memorandum 
not  go  through  the  PFE  evaluation  process.  He  said  that 
he  thought  it  was  "hogwash."  At  this  point  he  left  the 
room  and  the  panel  proceeded  with  its  evaluation. 

IV. 10. 11  The  Tenth  and  Eleventh  Factor  Team 

Meetings 

Commercial  Impact,  Reports  and  Reviews  (Factor  II) 

and  Background  (Factor  IV)  met  on  Thursday,  April  22,  at 

12:30  and  3 P.M.  respectively.  Factor  II  had  five  panel 

members  and  Factor  IV  had  seven  members.  These  factors 

were  rated  fairly  rapidly.  Some  of  the  discussion  referred 

back  to  points  brought  out  on  the  same  factors  under  the 
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agency  impact  evaluation.  There  tended  to  be  considerablv 
less  discussion  on  the  commercial  impact  evaluation  as  com- 
pared to  the  agency  impact  evaluation. 

IV. 10. 12  The  Twelfth  Factor  Team  Meeting 

The  final  day  of  panel  meetings  was  April  23.  The 
day  began  with  Commercial  Impact  Factor  VI,  Data  Collection 
Processes.  Five  panel  members  were  present  including  two 
new  panel  members.  After  the  evaluation  process  was  reviewed 
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This  same  panel  member  had  sent  a memorandum  to  the 
director  of  ETIP  on  April  16,  indicating  his  concern  over 
ETIP's  plans  to  commit  up  to  $2  million  for  the  projects 
and  stating  that  he  found  the  procurement  process  "overly 
complex  and  poorly  focused." 
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Two  of  the  three  bidders  on  the  commercial  impact 
contract  had  been  bidders  under  agency  impact. 


one  of  the  panel  members  expressed  a preference  for  discus- 
sing each  bidder  on  the  entire  factor  before  doing  any  rat- 
ings. Rating  sheets  for  all  of  the  sub-factors  were  turned 
in  and  about  15  minutes  was  taken  to  discuss  what  the  var- 
ious bidders  wrote  under  factor  VI.  The  rating  process 
then  proceeded  as  in  the  other  team  meetings. 

IV. 10. 13  The  Thirteenth  and  Fourteenth  Factor 

Team  Meetings 

The  final  two  factor  team  meetings  were  Commercial 
Impact  Factors  I,  Management  and  V,  Evaluation  Design. 

They  met  at  12:30  and  3 P.M.  with  four  and  five  panel 
members  respectively.  These  meetings  were  again 
relatively  short,  with  discussion  referring  back 
to  prior  team  meetings.  The  last  meeting  concluded 
at  about  4:15  p . m. 

IV. 11  Summarizing  the  Results  for  the  Contracting 
Office 

Following  the  last  factor  team  meeting  the  COTR  and 
the  author  began  the  summary  of  results  for  presentation 
to  the  contract  negotiator.  The  factor  scores  for  each 
bidder  were  multiplied  by  the  preassigned  factor  weights 
to  obtain  thp  overall  ratings.  The  scores  were  then  sum- 
marized in  tabular  forms  for  each  of  the  two  contracts  to 
obtain  a visual  display  of  the  factor  and  sub-factor  distri- 
butions. A determination  of  a recommended  competitive  range 


was  made  by  looking  at  both  the  final  overall  scores  and 

the  distribution  of  each  bidder's  scores  across  all  factors 

and  sub-factors.  It  was  felt  that  bidders  with  low  overall 

scores  or  bidders  which  had  acceptable  overall  scores  but 

had  an  unacceptable  number  of  factors  rated  at  four  or  below 

could  be  considered  unacceptable.  An  examination  of  the 

distribution  of  scores  revealed  that  on  the  Commercial  Im- 
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pact  contract  bidder  #101  clearly  dominated  the  other 
two  bidders.  In  fact  there  were  no  sub-factors  where  the 
scores  of  either  of  the  other  two  exceeded  bidder  #101. 

On  the  agency  impact  contract  the  situation  was  quite  dif- 
ferent. Here  three  of  the  six  bidders  were  rated  signifi- 
cantly higher  overall  then  the  other  three.  They  were  also 
quite  close  to  each  other  in  overall  score.  One  of  the 
three  more  favorable  bidders  on  the  agency  impact  contract 
was  the  same  bidder  which  dominated  on  the  commercial  impact 
contract.  However,  since  ETIP  had  stipulated  at  the  outset 
of  the  solicitation  that  it  would  not  award  both  contracts 
to  the  same  firm,  this  bidder  could  not  ultimately  be  con- 
sidered for  both  contracts. 

The  recommendation  which  the  COTR  was  prepared  to  make 
was  that  bidder  #101  be  given  further  consideration  as  the 
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The  bidders  will  be  referred  to  by  number  rather  than 
name.  These  numbers  are  also  used  with  the  data  presented 
in  figures  in  the  text  and  in  the  appendix. 


commercial  impact  contractor  and  that  bidders  #202  and  #203 
be  given  further  consideration  as  agency  impact  contractors 
In  this  case  further  consideration  would  require  clarifica- 
tion on  sub-factors  where  various  factor  teams  had  raised 
issues  for  clarification  with  each  bidder,  or  where  the 
bidder  was  rated  below  a seven. 

In  a May  4,  1976  memorandum,  the  COTR  conveyed  these 
'recommendations  to  the  contract  negotiator.  He  provided 
a chart  (Figure  IV. 14)  giving  the  scores  and  then  provided 
seven  pages  of  narrative  to  summarize  the  issues  on  which 
each  bidder  had  problems.  (COTR,  Note  91).  The  narrative 
followed  the  break  points  set  out  on  the  factor/sub-factor 
rating  sheet.  A bidder  was  said  to  be  acceptable  or  better 
on  factors  where  their  score  was  seven  of  greater  and  unac- 
ceptable on  factors  where  the  score  was  less  than  seven. 

In  elaborating  further,  a bidder  was  said  to  have  moderate 
problems  on  issues  related  to  sub-factors  where  the  agreed 
upon  score  was  six,  more  serious  problems  where  the  score 
was  five,  and  very  serious  problems  where  the  score  was 
four  or  below  (COTR,  Note  92).  As  an  example,  the  section 
from  the  memo  on  bidder  #206  read  as  follows: 

Bidder  #206 

The  ETIP  panel  rated  #206  as  acceptable  on  only 
two  of  the  seven  proposal  evaluation  factors. 

#206  was  considered  unacceptable  on: 
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o Management 

o Reports  and  Reviews 

o Background  Information 

0 Evaluation  Design 

o Data  Collection  Process 

Among  the  issues  with  which  #206  had  moderate 
problems  were: 

o Its  evaluation  management  capabilitv 

o Its  understanding  of  and  capability  to 
define  the  detailed  evaluation  objectives 

o Its  ability  to  anticipate  the  incorporation 
of  the  results  of  Phase  One  in  the  restated 
objectives  for  Phase  Two 

Among  the  issues  with  which  #206  had  more  serious  prob- 
lems were: 

o Its  understanding  of  the  soecific  objectives 
of  the  program(s)  and/or  exoer iment ( s ) 
that  are  to  be  evaluated 

o Its  ability  to  design  the  overall  data 
collection  process 

o Its  ability  to  design  the  overall  data 
analysis  process 

Among  the  issues  with  which  #206  had  very  serious 
problems  were: 

0 Its  ability  to  coordinate  its  activities 
with  those  of  other  related  individuals 
and  organizations  not  specifically  referred 
to  in  section  II 

o Its  understanding  of  the  programs,  organiza- 
tion policies  and  procedures  of  other 
relevant  organizations. 

This  gave  the  contract  negotiator  considerably  more 
information  than  was  available  in  the  final  numerical  score 
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The  final  ratings  are  based  on  the  following  factor  ratings 
assigned  by  the  ETIP  panel,  and  by  the  weights  established 
at  the  outset  of  the  solicitation: 
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FIGURE  IV. 14  SUMMARY  DISPLAY  OF  SCORES  FOR  THE 
CONTRACT  NEGOTIATOR 
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and  in  a manner  which  was  consistent  across  all  bidders. 

In  this  way  the  important  issues  for  use  in  clarification, 
negotiation  and  debriefing  were  not  obscured  by  the  numbers. 


IV. 12  Preparing  for  Orals 

Preparation  for  face-to-face  discussions  with  the  three 
remaining  bidders  was  accomplished  by  the  COTR,  the  contract 
negotiator  and  several  ETIP  staff  through  a review  of  the 
notes  on  issues  for  clarification  which  viere  raised  by  the 
factor  teams.  That  is,  on  any  sub-factor  where  the  bidder 
was  rated  at  six  or  below  or  where  the  panel  had  otherwise 
requested  a specific  clarification,  the  COTR  went  back  to 
the  notes  of  the  evaluation  process  to  prepare  questions 
to  initiate  the  discussions.  At  least  one  question  was 
written  for  each  sub -factor  where  the  nan el  required  clarifi- 
cations. During  this  time  period,  telephone  calls  were 
placed  to  people  referenced  by  the  bidders  in  their  propo- 
sals as  being  familiar  with  their  past  performance. 

With  the  factor  team  meetings  completed,  the 
bidders'  cost  proposals  were  provided  for  review  by  both 
the  COTR  and  a cost  analyst  in  the  contracting  office. 

There  was  a considerable  difference  in  the  costs  proposed 
by  the  two  remaining  bidders  on  the  agency  impact  contract. 
The  cost  difference  was  accounted  for  by  a number  of  factors 
including,  primarily,  different  total  man  months  of  effort 


proposed,  different  costs  for  direct  and  indirect  labor 
(benefits),  overhead,  and  G&A  (General  and  Administrative), 
and  different  travel  and  secretarial  costs. 

The  task  which  remained  for  the  COTR  was  to 
improve  the  estimates  of  each  bidder’s  actual  sub- 
factor score  (i.e.,  decrease  the  range  of  uncertainty),, 
obtain  clarification  of  cost  estimates  and  consider  all 
of  the  information  together  to  make  a determination  of  which 
offer  appeared  to  be  in  the  best  interest  of  the  government 
(cost  and  other  factors  considered).  A meeting  was  held 
with  the  contract  negotiator  on  May  12  (COTR,  Note  93)  to 
plan  the  process  for  this  review.  It  was  decided  to  invite 
each  bidder  in  for  three  hours.  They  would  be  allowed  to 
make  a brief  presentation  on  their'  proposal,  followed  by 
a question  and  answer  period  on  both  cost  and  technical 
information.  All  meetings  would  be  tape  recorded. 

The  bidders  were  called  by  the  contract  negotiator 
and  told  that  ETIP  wished  to  meet  with  them  for  face-to- 
face  clarifications  of  some  issues  in  their  proposals. 

They  were  told  which  sub-factors  these  issues  related  to 
so  that  they  could  prepare  for  the  meeting. 

IV. 13  Orals  and  Written  Confirmations 

Bidder  #203  was  invided  to  appear  at  the  Department 
of  Commerce  the  morning  of  May  25,  1976.  They  arrived  with 


six  members  of  their  oroposed  team,  including  the  project 
manager  and  various  technical  area  leaders. 

Following  initial  introductions  the  project  manager 
was  given  the  ooportunity  to  make  a brief  presentation  on 
the  organization's  proposal.  He  responded  by  discussing 
the  characteristics  of  the  organization  including  its  size, 
the  technical  skills  of  its  staff,  and  some  of  its  areas 
of  prior  work.  He  said  that  in  responding  to  the  format 
of  the  RFP  they  were  unable  to  bring  out  a comprehensive 
illustration  of  how  they  might  deal  with  an  experiment  and 
hoped  to  do  so  during  the  discussions.  The  meeting  then 
proceeded  with  a dialogue  surrounding  the  specific  questions 
which  ETIP  had  prepared  (Tape  of  meeting.  Note  9^). 

Questions  were  asked  about  both  management  and  techni- 
cal aspects  of  the  project.  The  management  questions  ranged 
from  asking  which  senior  management  people  would  be  full 
time  on  the  project  to  asking  about  the  internal  process 
for  resolving  conflict  among  people  on  their  team.  The 
technical  dialogue  included  questions  on  who  wrote  specific 
portions  of  the  proposal  to  discussion  about  what  the  appro- 
priate research  techniques  would  be  for  specific  evaluations 
under  various  circumstances  (Tape  of  Meeting,  Note  95). 

Bidder  #202  was  invited  to  appear  on  the  afternoon 
of  May  25.  They  brought  four  members  of  their  team  includ- 
ing the  project  manager,  one  of  the  technical  group  leaders 


and  two  technical  staff  members. 


Following  introductions,  the  project  manager  described 
the  process  which  was  used  in  preparing  their  proposal  and 
discussed  in  very  general  terms  what  their  organization 
would  do  in  carrying  out  the  work.  He  also  indicated  that 
they  had  some  questions  concerning  the  extent  and  nature 
of  the  involvement  of  the  ETIP  staff  in  the  project.  The 
meeting  proceeded  with  discussion  related  to  the  issues 
on  which  the  evaluation  panel  suggested  clarifications. 

(Tape  of  meeting,  Note  96). 

Questions  covered  both  management  and  technical  aspects 
of  the  project.  Management  questions  included  asking  about 
costing  of  various  aspects  of  the  proposal  relative  to  the 
work  to  be  done  and  asking  about  the  process  of  integrating 
various  technical  specialists  into  the  work  as  they  are 
required.  Technical  discussion  ranged  from  asking  where 
specific  expertise  was  within  their  team  to  questions  of 
what  specific  technical  methods  would  be  appropriate  for 
various  potential  types  of  evaluations  (Tape  of  meeting. 

Note  97). 

Bidder  #101  was  invited  to  appear  the  morning  of  May 
26:  They  brought  six  people  to  the  meeting  including  a 

representative  of  company  management,  the  project  supervisor, 
the  project  leader,  technical  group  leaders,  and  technical 


staff. 


The  project  supervisor  introduced  those  present  and 
described  the  corporate  commitment  to  the  project.  He  then 
turned  the  presentation  over  to  the  project  leader  who  dis- 
cussed what  bidder  //lOl  saw  as  some  of  the  key  issues  of 
the  work  and  how  they  generally  intended  to  deal  with  these 
issues.  This  was  followed  by  a dialogue  on  the  issues  which 
required  clarification.  (Tape  of  meeting,  Note  98). 

Again,  this  covered  both  management  and  technical  mat- 
ters. Management  questions  included  asking  about  the  level 
of  commitment  that  specific  people  would  make  to  the  project 
and  asking  about  how  resource  allocation  decisions  would 
be  made.  Technical  issues  tended  to  center  around  the  pro- 
cess for  developing  and  finding  a home  for  the  commercial 
impact  evaluation  system  (Tape  of  meeting.  Mote  99). 

Following  the  face-to-face  meetings  the  COTR  sent  the 
contract  negotiator  written  lists  of  questions  and  state- 
ments for  each  of  the  three  bidders.  This  was  to  be  used 
to  obtain  written  confirmation  of  positions  taken  by  the 
bidders  or  observations  made  by  the  ETIP  staff  which  partici- 
pated in  the  oral  discussions..  The  approach  taken,  as  indi- 
cated in  the  memo  transmitting  the  questions  to  the  contract 
negotiator,  was  to  present  to  the  bidders  with  both: 

o A standard  concerning  where  ETIP  felt  the  bid- 
der should  be  with  respect  to  understanding, 
approach  or  capability  along  a single  or  par- 
ticular set  of  factors  or  sub-factors,  and 
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o ETIP’s  observations  as  to  where  the  bidder 
actually  was  with  respect  to  this  standard 
(COTR,  Note  100). 

In  other  words,  the  purpose  was  to  indicate  clearly  where 
ETIP  felt  that  the  bidder  still  required  improvement  his 
position. 

For  example,  one  of  the  questions  was: 

The  commercial  impact  contract  requires  knowl- 
edge and  experience  in  the  area  of  economic 
analysis  at  the  firm  level  of  aggregation. 

It  is  important  for  the  contractor  to  be  able 
to  obtain  access  to  data  which  is  not  generally 
publicly  available.  The  contractor  must  also 
be  able  to  analyze  this  data  from  economic 
perspective.  Your  proposal  does  not  evidence 
this  capability.  Please  comment  on  this  in 
terms  of  past  experience  of  the  individuals 
who  are  proposed  for  this  project  with  respect 
to  subfactors  VI. 2,  VI. 3,  VII, 1.  Limit  your 
response  to  1 page  (COTR,  Note  101). 

The  questions  were  sent  out  by  the  contracting  office 
on  June  4,  and  responses  were  requested  in  by  the  close 
of  business  June  11,  1976.  Bidder  #203  was  sent  three  ques- 
tions, bidder  #202  was  sent  nine  questions,  and  bidder  101 
was  sent  one  question. 


IV. 14  Summary  of  Results  and  Award 

The  COTR  and  two  other  panel  members  reviewed  the  writ- 
ten responses  from  each  of  the  thre*e  bidders,  together  with 
the  records  of  the  face-to-face  meetings  and  the  findings 
of  the  technical  evaluation  team.  For  each  sub-factor  where 
an  issue  had  been  re-examined  a more  precise  rating  was 
assigned  by  this  team  based  on  the  clarifications  and  ad- 


ditional  information  which  were  provided  by  the  bidders. 

The  overall  results  were  that  bidders  #203  and  #101  were 
both  considered  acceptable  on  all  seven  prooosal  evaluation 
factors  while  bidder  #202,  considered  unacceptable  on  three 
factors  after  the  panel  evaluation,  was  now  considered  unac 
ceptable  on  an  additional  three  factors.  The  COTR ’ s deci- 
sion was  therefore  to  recommend  that  bidder  #101  be  nego- 
tiated with  for  the  commercial  impact  evaluation  contract 
and  that  bidder  #203  be  negotiated  with  for  the  agency  im- 
pact evaluation  contract  (COTR,  Note  102). 

These  recommendations  were  conveyed  to  the  contract 
negotiator  in  a memorandum  on  June  17,  1976.  The  form  of 
this  memorandum  followed  the  narrative  form  of  the  COTR ' s 
May  4 memorandum,  i.e.,  discussion  of  acceptability  based 
on  factor  scores  of  seven  or  greater  and  unacceptability 
based  on  factor  scores  below  seven  with  elaboration  of  mode 
ate,  more  serious  and  very  serious  problem.s  based  on  the 
specific  sub-factor  scores. 

The  COTR  sent  a second  memorandum  to  the  contract  nego 
tiator  describing  additional  information  which  needed  to 
be  considered  during  final  negotiations.  This  included: 

o advising  that  the  total  sum  of  both  contracts 
could  not  exceed  $1.8  million  and  that  it 
would  be  technically  appropriate  to  award 
a $.9  million  contract  to  bidder  #101  and 
a $.9  million  dollar  contract  to  bidder  #203* 
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o suggesting  a method  for  negotiating  on  the 
sub-factor  issue  where  bidder  #203  continued 
to  have  a moderate  problem,  and 

0 suggesting  obtaining  full  time  commitments 
from  two  individuals  on  bidder  #203 ’s  team 
(COTR,  Note  103). 

Best  and  final  offers  were  requested  on  June  23,  and 
received  by  telecopy  on  June  24.  They  were  reviewed  by 
The  COTR  and  on  June  25  he  sent  memos  to  the  contract  nego- 
tiator indicating  that  he  felt  the  quality  and  quantity 
of  effort  proposed  to  be  reasonable  and  necessary  for  execu- 
tion of  the  projects  (COTR,  Note  104,  Note  105). 

The  contracting  office  then  requested  a written  indica- 
tion of  the  shifts  which  resulted  in  bidder  #202’ s sub-factor 
ratings  based  on  additional  information  which  was  provided 
from  the  oral  and  written  clarifications.  This  listing 
was  prepared  by  the  COTR  and  the  author  and  sent  to  the 
contracting  office  and  June  28  (COTR  & Libman,  Note  106). 

Contracts  dated  June  30,  1976  were  signed  with 
both  bidder  #101  and  #203  and  work  on  the  projects 
began  during  the  second  week  of  July. 

IV. 15  Debriefing 

On  July  9,  the  COTR  and  the  author  sent  a memorandum 
to  those  who  participated  on  the  evaluation  panel.  The 
process  which  followed  the  evaluation  panel  meetings  and 
its  results  were  described  and  panel  members  were  thanked 
for  their  participation  (COTR  & Libman,  Note  107). 


The  contracting  office  received  one  request  for  a de- 
briefing. This  was  from  bidder  #202.  The  COTR,  the  author 
and  the  contract  negotiator  met  with  three  representatives 
of  bidder  #202  on  August  4,  1976.  The  initial  discussion 
centered  around  the  factors  on  which  bidder  #202  was  found 
to  be  less  than  acceptable  (COTR,  Note  108).  The  dialogue 
and  tone  of  the  meeting  indicated  that  bidder  #202  disagreed 
considerably  with  the  evaluation  of  their  technical-  capa- 
bility and  were  angry  about  losing  when  they  considered 
themselves  more  qualified  than  the  winner  (COTR,  Note  109). 
However,  they  did’not  file  a formal  protest. 


IV. 16  Summary  and  Transition  to  Chapter 

This  chapter  has  provided  a chrono 
ted  to  the  procurement  of  two  evaluatio 
by  the  Experimental  Technology  Incentiv 
tion  was  presented  on  issues  which  part 
to  the  process  and  on  how  they  were  res 
the  actual  evaluation  process  were  pres 
and  anecdotes  were  related  to  give  the 
of  how  the  process  resulted  in  ratings 
review  and  summary  process  was  describe 
process  progressed  from  the  evaluations 
the  signing  of  contracts. 

The  next  two  chapters  will  evaluat 


_V 

loey  of  events  rela- 
n systems  contracts 
es  Program.  Informa- 
icipants  brought 


olved 

Data 

f rom. 

ented 

in  figures 

reade 

r a picture 

for  b 

idders . 

The 

d to 

show  how 

the 

by  the  panel 

to 

e the 

results 

of 

using  PFE  against  some  of  the  issues,  which  participants 


in  the  procurement  raised.  The  results  will  also  be  evalu- 
ated against  some  of  the  issues  which  are  raised  by  the 
literature  which  was  reviewed  in  Chapter  II.  The  descrip- 
tion presented  in  this  chapter  and  other  data  from  the  pro- 
cess located  in  the  appendix  will  be  used  in  this  evaluation. 
This  information  is  presented  to  enable  the  reader  to  per- 
form his  own  independent  evaluation. 
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CHAPTER  V 

EVALUATION/CRITIQUE 


V . 1 Introduction 

The  purpose  of  this  chapter  is  to  evaluate  the  outcome 
of  ETIP’s  use  of  the  Parametric  Factor  Evaluation  process. 

It  presents  an  assessment  of  how  well  the  PFE  process  worked 
in  solving  many  of  the  critical  issues  which  ETIP  faced 
during  its  source  selection.  The  issues  for  this  evaluation 
are  those  which  were  raised  by  participants  in  the  ETIP 
project,  together  with  a selection  of  issues  from  the  litera- 
ture which  the  author  felt  were  applicable  to  ETIP*s  prob- 
lem. 


The  format  for  presenting  the  evaluation  is  to  provide 
some  general  discussion  of  the  issues  and  procedures  for 
evaluation,  to  discuss  the  outcome  for  each  selected  issue 
individually,  and  finally  to  summarize  the  results  of  ETIP's 
use  of  PFE  in  terms  of  successes,  failures,  and  uncertain- 
ties. The  general  discussion  of  the  issues  indicates  how 
an  initial  set  of  issues  were  accumulated  and  placed  into 
categories  and  how  a specific  set  of  issues  were  singled 
out  for  in-depth  discussion.  The  technique  for  gathering 


additional  data  from  participants  in  the  source  selection 
is  described  and  the  parameters  which  may  affect  the  inter- 


pretation  of  the  results  are  discussed.  The  evaluation 
of  each  specific  issue  contains  a conclusion  about  how  suc- 
cessful PFE  was  in  dealing  with  the  issue  and  presents  the 
information  which  the  author  used  in  reaching  this  conclu- 
sion. Finally  the  summary  of  the  evaluation  is  presented 
in  which  the  issues  are  placed  into  categories  of  those 
for  which  PFE  appears  to  present  a successful  solution, 
those  which  PFE  appears  to  be  unsuccessful  at  solving,  and 
those  for  which  a conclusion  cannot  be  confidently  made 
using  the  data  available.  An  indication  of  the  points  at 
which  ETIP  departed  from  the  process  as  it  was  originally 
used  at  Wright  Field  and  a description  of  changes  which 
ETIP  made  in  its  own  subsequent  applications  of  the  process 
are  presented  as  part  of  Chapter  VI. 

V . 2 The  Issues,  Additional  Data,  and  Parameters 

V.2.1  Listing  of  Issues 

The  issues  presented  in  this  section  provide  the  basis 
for  evaluating  the  effectiveness  of  the  PFE  process  in  sol- 
ving ETIP’s  source  selection  problem.  The  list  of  issues, 
which  is  presented  in  Figure  V.1,  was  drawn  by  two  methods. 
The  first  was  by  examining  the  records  of  ETIP’s  use  of 
the  PFE  process  for  questions  which  were  raised  before, 
during,  and  after  the  source  selection  by  the  people  who 
were  involved.  The  second  was  by  writing  down  issues  the 
author  came  upon,  through  his  readings,  discussions-  with 
others,  or  reflection  on  the  problem,  which  seemed  appli- 
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cable  to  ETIP’s  situation.  Comprehensiveness  in  covering 
the  major  issues  rather  than  independence  among  the  issues 
was  the  objective.  In  fact,  it  is  partly  the  underlying 
interrelatedness  among  these  issues  which  makes  system  source 
selection  a difficult  problem.  The  listing  of  issues  is 
not  meant  to  imply  that  they  are  necessarily  of  equal  impor- 
tance. Also,  issues  of  significance  to  some  source  selec- 
tions but  not  directly  relevant  to  ETIP’s  problem  are  not 
covered  here. 

V.2.2  Categories  of  Issues 

When  a preliminary  listing  of  issues  was  accom.pl ished , 
the  list  turned  out  to  be  quite  long.  In  order  to  provide 
some  structure  to  this  evaluation,  a grouping  was  sought. 

Each  issue  was  sorted  into  a group  with  the  other  issues 
to  which  it  seemed  most  closely  related.  As  this  was  being 
done,  the  common  elements  within  groups  emerged  and  headings 
were  assigned.  A statement  was  written  for  each  heading 
to  try  to  illustrate  the  common  elements.  Issues  were  then 
moved  among  these  groupings  if,  upon  re-examination,  they 
seemed  like  they  fit  better  elsewhere.  As  a few  new  issues 
were  later  discovered,  they  were  placed  in  one  of  the  exist- 
ing groups.  In  some  cases  an  issue  could  have  fit  under 
one  or  more  other  headings  and  was  assigned  somewhat  arbi- 
trarily. For  evaluation  purposes  the  issue  is  more  impor- 
tant than  the  major  category  under  which  it  is  grouped. 
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1 . 
2. 

3. 

4. 

5. 

6 . 


7. 


8. 

9. 

10. 
1 1 . 
12. 
13. 


ISSUE  CATEGORIES 
Effective  Communications 

Sensitivity  and  Selectivity  in  Obtaining  Bids 

Assessment  of  Bidders’  Capability 

Use  of  Experts  in  the  Evaluation  of  Proposals 

The  Panel’s  Role  in  Evaluation  of  the  Proposals 

Equity  of  Treatment  Across  Bidders 

Risk  Associated  with  Statement  of  Work 

Basis  for  Award 

Traceability  of  the  Process 

Relationship  with  the  Contracting  Office 

Resource  Requirements 

Education  of  Government  Personnel 

Education  of  Bidders 


FIGURE  V.1 


ISSUES  AND  ISSUE  CATEGORIES 


1 . EFFECTIVE  COMMUNICATIONS 


Did  the  format  and  content  requirements  of  the  PFE  process 
provide  for  effective  communications  between  the  government 
and  potential  bidders  and  between  bidders  and  the  STIP  propo- 
sal evaluation  panel? 

Format 

+ Did  PFE  provide  too  much  structure  for  the  bidders’ 
response? 

Is  the  bidder  able  to  display  his  competence? 

Does  it  present  problems  for  the  bidder  or  the  panel 
when  no  formal  structure  is  used  for  the  object 
of  the  procurement? 

+ Did  the  panel  have  difficulty  evaluating  the  proposals 
on  a sub-factor  basis? 

Was  the  panel  able  to  find  the  information  it  needed? 
Did  they  feel  that  the  evaluation  was  being  done 
out  of  context  from  the  project? 

Was  the  RFP  poorly  organized? 

Was  the  RFP  confusing? 

Was  the  existence  of  two  statement s-of-work  confus- 
ing? 

Was  the  RFP  too  long?  Should  its  size  have  been  lim- 
ited? 

+ Should  the  size  of  proposals  have  been  limited? 

(Did  length  correlate  with  final  scores?) 

+ Did  the  bidders  understand  what  the  evaluation  panel 
looked  at  in  making  their  evaluations?  Did  knowing 
this  help? 

Did  the  briefings  allow  for  adequate  questioning  by 
the  prospective  bidders?  (differences  in  information 
among  bidders) 

+ Did  the  process  enable  the  panel  to  deal  effectively 
with  differences  in  uncertainty  in  information  among 
bidders  during  the  evaluation? 

Did  the  question  form  provide  an  effective  mechanism 
for  obtaining  missing  information  and  clarification- 
after  the  proposals  were  received? 

+ indicates  that  the  issue  is  explicitely  discussed  in 
Section  V.3 
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Content 


+ Did  the  process  communicate  to  potential  bidders  what 
it  was  the  government  wanted  to  buy? 

(Did  bidders  know  what  they  were  bidding  on?) 

+ Did  the  process  communicate  to  prospective  bidders 

the  skills  which  would  be  required  to  do  the  work  on 
the  project? 

Did  the  process  succeed  in  explaining  to  bidder  the 
reasons  why  it  was  being  used? 

Was  information  provided  to  bidders  on  the  history 
of  the  PFE  process  harmful  or  useful  or  unnecessary 

+ Were  many  of  the  sub-factors  not  useful  to  the  bidders 
or  the  panel  (e.g.,  Management  or  Reports  & Reviews 
sub-factors)? 

Was  too  much  detail  required  from  the  bidders? 

(Information  which  was  not  useful  for  effective 
communication.)  Does  the  process  encourage  gold 
plating  of  proposals? 

Did  the  process  make  the  priorities  of  the  government 
clear? 

Did  the  process  enable  the  government  to  describe  as- 
pects of  the  project  for  which  few  details  were  known 
as  well  as  those  for  which  there  was  a great  deal  of 
detail? 

Were  problems  caused  by  the  requirement  for  the  bidder 
to  respond  to  issues  which  are  traditionally  thought 
to  be  the  responsibility  of  the  government?  (e.g., 
how  to  rationalize  multiple  and  conflicting  objectives 
of  the  project ) . 

+ Was  parroting  back  of  information  from  the  RFP  by  bid- 
ders to  the  exclusion  of  other  information  a problem 
for  the  panel? 

2.  SENSITIVITY  AND  SELECTIVITY  IN  OBTAINING  BIDS 

Was  ETIP  successful  in  obtaining  an  acceptable  number  of 
responsive  bids? 


FIGURE  V.1  Continued 
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+ 


+ 


Were  enough  responsive  bids  received  to  allow  for  a 
confident  evaluation? 


Did  the  process  enable  prospective  bidders  to  self- 
select effectively  on  bidding? 

Were  a large  number  of  non-respons i ve  bids  received? 
Did  a lot  of  qualified  firms  not  bid? 

Did  the  process  help  the  bidder  to  know  which  people 
to  use  in  responding  to  the  RFP? 

3.  ASSESSMENT  OF  BIDDERS'  CAPABILITY 


Did  PFE  enable  ETIP  to  assess  whether  bidders  had  the  re- 
quired capabilities  when  there  was  no  track  record  in  con- 
tracting organizations,  i.e.,  of  having  a large  number  of 
people  with  the  diverse  skills  and  interests  which  were 
required  here  to  work  together  over  a long  period  of  time 
to  develop  the  evaluation  systems. 


+ Did  PFE  provide  for  an  effective  evaluation  of  manage- 
ment capability? 

+ Did  PFE  provide  for  an  effective  evaluation  of  techni- 
cal capability? 

+ Did  PFE  anticipate  the  problems  which  were  known  to 

exist  in  doing  evaluations  and  in  developing  systems? 
Did  the  criteria  provide  for  an  assessment  of  the 
relationship  between  what  was  proposed  and  later 
performance? 

Was  the  award  based  upon  proposals  which  would  hav 
some  relationship  to  the  approach  subsequently  use 
Was  the  material  provided  by  the  successful  contrac- 
tors useful  or  not  to  the  PAAs? 

4.  USE  OF  EXPERTS  IN  THE  EVALUATION  OF  PROPOSALS 

Was  ETIP  able  to  obtain  and  make  use  of  experts  in  the  eval- 
uation of  proposals? 


+ Was  it  difficult  to  get  experts  (specialist  and  those 
with  prior  experience  with  prospective  bidders)  to 
participate  when  there  was  a large  RFP  and  the  propo- 
sals were  expected  to  be  large? 

Was  expert  information  useful  in  the  evaluation?  Did 
the  panel  always  follow  the  advice  of  experts? 

FIGURE  V.1  Continued 
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How  difficult  was  it  to  identify  people  to  put  on  the 
evaluation  team?  How  difficult  was  it  to  train  them 
in  the  evaluation  process?  How  difficult  was  it  to 
motivate  them  to  do  a good  job? 

5.  THE  PANEL'S  ROLE  IN  EVALUATION  OF  THE  PROPOSALS 

Did  the  Factor  Team  process  work  effectively  in  obtaining 

agreement  on  ratings  on  all  sub-factors  for  all  bidders? 

+ Did  panel  members  who  agreed  to  participate  actually 
show  up? 

How  difficult  was  it  to  schedule  the  meetings? 

+ Were  panel  members  able  to  -arrive  at  initial  ratings 
prior  to  factor  team  meetings? 

+ Was  the  panel  able  to  arrive  at  final  scores  without 
voting? 

Was  the  panel  able  to  deal  effectively  with  strong 
individual  disagreements? 

+ Did  the  panel  members  realize  how  their  individual 
positions  affect  the  final  score,  i.e.,  do  they  see 
other  panel  members  modifying  their  position  in  response 
to  the  discussion. 

Do  panel  members  understand  how  the  final  score 
i?s  reached? 

+ Do  the  final  scores  differ  significantly  from  the  means 
of  the  Initial  scores  of  the  individual  panel  members? 

+ Were  the  panels  ratings  and  recommendations  used  after 
the  panel  meetings  were  concluded? 

+ What  impact  did  those  with  extreme  scores  or  greater 

variability  in  their  ratings  have  on  the  final  scores? 

What  affected  the  time  to  reach  agreement? 

The  distribution  of  initial  scores? 

The  specific  panel  members  present? 

+ How  long  did  it  take  to  reach  agreement  on  factor  scores 
after  the  sub-factor  scores  were  agreed  to? 

+ Were  ’’junior”  panel  members  able  to  affect  the  score 
if  their  bosses  were  in  the  room  and  their  initial 
scores  differed? 
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Did  panel  members  read  the  correct  parts  of  the  propo- 
sals in  arriving  at  their  ratings? 

6.  EQUITY  OF  TREATMENT  ACROSS  BIDDERS 

Did  the  process  provide  for  a fair  and  equitable  treatment 

of  all  of  the  bids? 

+ Did  the  panel  deal  effectively  with  the  biases  of  its 
members? 

+ ^ Did  the  criteria  used  in  evaluation  provide  for  a con- 

sistent evaluation  across  proposals?  Were  the  criteria 
applied  consistently  across  prooosals? 

Were  any  problems  caused  due  to  the  lack  of  a require- 
ment on  the  bidders  to  respond  to  all  of  the  illustra- 
tive questions? 

+ Did  the  lack  of  restriction  on  admissible  information 
lead  to  any  inequity  in  the  process? 

+ Did  evaluation  of  the  proposals  while  knowing  the  iden- 
tity of  the  bidders  create  any  problems  - biases  in 
the  final  ratings,  protests? 

7.  RISK  ASSOCIATED  WITH  THE  STATEMENT  OF  WORK 

Did  PFE  enable  ETIP  to  deal  effectively  with  the  "added 

risk"  associated  with  a flexible  statement  of  work. 

+ Was  ETIP  able  to  write  a statement  of  work  which  would 
spell  out  deliverables  and  yet  avoid  numerous  contract 
changes  which  generally  occur  in  situations  where  re- 
quirements, policies,  and  the  state  of  the  art  are 
changing? 

Did  ETIP  management  accept  it? 

Did  the  contracting  officer  accept  it? 

+ Did  the  flexible  statement  of  work  scare  off  bidders? 

+ V/as  ETIP  able  to  assess  the  commitment  of  the  bidders' 

top  management  to  devote  the  necessary  resources  to 
the  project? 

Was  ETIP  able  to  determine  whether  it  was  within  the 
longer  term  interest  of  management  to  develop  the  de- 
sired systems? 
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Did  PFE  deal  with  the  lack  of  incentive  for  cost  con- 
trol in  CPFF  contracts? 

Would  PFE  allow  ETIP  to  alter  the  timing  and  level 
of  effort  of  the  project  after  the  contract  was  signed? 

8.  BASIS  FOR  AWARD 

9 

Did  use  of  PFE  provide  a basis  for  discriminating  among 
bidders  on  criteria  other  than  price  in  determining  who 
to  negotiate  with? 

+ Did  PFE  provide  for  overall  technical  discrimination 
among  bidders?  Was  there  a sufficient  distribution 
in  the  final  scores  to  determine  a competitive  range? 

+ Did  PFE  provide  a basis  for  discriminating  among  bidders 
on  specific  technical  issues?  i.e.,  Did  the  specific 
issues  get  washed  out  by  the  overall  technical  score? 

How  much  of  a factor  was  schedule  in  the  final  deci- 
sion? 

Did  PFE  provide  a capability  for  meeting  requirements 
to  allow  a bidder  to  improve  his  proposal? 

+ Was  technical  leveling  a problem? 

+ Was  technical  transfusion  a problem? 

Did  the  process  provide  the  contract  negotiator  with 
the  information  he  needed  to  negotiate  a contract  award? 

+ How  much  of  a factor  was  bid  price  in  the  final  deci- 
sion? Did  the  procurement  turn  out  to  be  a price  com- 
petition? Did  the  process  provide  a capability  for 
dealing  with  potential  buy-ins? 

9.  TRACEABILITY  OF  THE  PROCESS 

Are  the  results  of  the  evaluation  traceable  so  that  an  inde- 
pendent examination  of  the  record  would  reconstruct  the 
bases  for  the  final  selections? 

+ Did  the  process  communicate  that  there  was  no  sole 
source  position?  Can  the  process  be  fixed? 

+ Did  bidders  understand  how  the  decisions  on  who  wins 
actually  get  made? 
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+ Can  changes  in  scores  be  traced  between  the  initial 
ratings  of  panel  members  and  final  panel  ratings  and 
between  final  panel  ratings  and  the  award  decisions? 

-I-  Did  the  panel  follow  the  written  procedure? 

+ Did  leaving  the  relative  weightings  of  the  sub-factors 

up  to  the  teams  lead  to  any  problems  (e.g.,  appearance 
of  inequities)? 

10.  RELATIONSHIP  WITH  THE  CONTRACTING  OFFICE 

Did  the  contracting  office  feel  vulnerable  to  award  protests 

by  using  PFE? 

+ Is  PFE  legal  under  the  Federal  Procurement  Regulations? 

+ Did  the  contracting  officer  feel  that  the  selections 

made  by  using  the  process  would  be  defensible  if  there 
were  a GAO  investigation? 

+ Did  the  contract  negotiator  accept  the  evaluation  of 
the  panel?  Did  he  want  the  panel  to  revise  their  re- 
port? Did  he  want  another  level  of  technical  review 
of  the  panel’s  evaluation? 

+ Did  using  PFE  result  in  timely  awards? 

+ Were  there  any  protests  of  the  award? 

+ Were  any  debriefings  requested? 

11.  RESOURCE  REQUIREMENTS 

What  level  of  resources  over  what  interval  of  time  were 

required  to  effect  the  procurements? 

+ How  much  time  was  required  to  do  system  or  evaluation 
design  work  prior  to  issuing  the  RFP? 

o elapsed 

o total  time  of  various  people 

How  much  time  did  it  take  to  write  the  RFP? 

o elapsed 

o total  time  of  various  people 

+ How  much  time  did  it  take  for  bidders  to  respond  to 
the  RFP?  What  was  the  volume  of  material  submitted 
(as  a surrogate  for  cost  of  responding)? 
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+ How  much  time  did  it  take  for  the  panel  to  evaluate 

the  proposals?  How  much  time  did  it  take  to  read  the 
proposals?  How  much  time  did  the  panel  meetings  (and 
preliminary  meetings)  take?  How  much  time  would  oral 
presentations  have  taken? 

+ How  long  did  it  take  to  make  an  award  once  the  panel's 
evaluations  were  complete? 

How  much  time  did  it  take  to  write  up  the  results 
of  the  panel  meetings? 

How  long  did  it  take  to  prepare  for  oral  clarifica- 
tions? How  long  did  it  take  to  request  written 
confirmations  of  oral  clarifications?  How  long 
did  it  take  to  reach  a decision  once  the  written 
clarifications  were  received? 

+ How  long  did  it  take  to  prepare  for  the  debriefing? 

Would  the  same  decision  have  been  made  by  using  another 
method?  Does  another  method  deal  as  effectively  with 
the  same  issues  for  less  effort? 

12.  EDUCATION  OF  GOVERNMENT  PERSONNEL 

Did  use  of  PFE  result  in  education  of  government  personnel 

who  would  be  involved  in  the  evaluation  system  development 

projects? 

+ Did  use  of  PFE  result  in  education  of  government  person- 
nel so  that  they  would  be  able  to  effectively  evaluate 
the  bidders’  proposals? 

+ Did  use  of  PFE  result  in  education  of  government  person- 
nel so  that  they  would  be  better  able  to  work  with 
or  manage  the  resulting  contractors? 

Were  program  participants  willing  to  be  involved 
in  the  procurements? 

13.  EDUCATION  OF  BIDDERS 

Did  use  of  PFE  result  in  education  of  bidders  useful  to 

their  future  interactions  with  ETIP? 

+ Did  use  of  PFE  result  in  training,  learning,  and  educa- 
tion of  the  successful  bidders  which  was  useful  for  their 
performance  on  the  contracts? 
o early  performance 
o later  performance 

Did  use  of  PFE  result  in  education  of  unsuccessful 
bidders  which  was  useful  in  bidding  on  other  ETIP  RFPs? 

FIGURE  V.1  Continued 


The  categories  serve  mostly  as  a structure  for  the  discus- 
sion which  follows  in  Section  V.3* 

V.2.3  Selection  of  Specific  Issues  for  Discussion 
Equal  treatment  of  all  of  the  issues  was  thought  to 
be  neither  desirable  nor  possible  within  the  scope  of  this 
case  study.  It  was  therefore  necessary  to  make  some  deci- 
sions concerning  depth  of  evaluation  for  each  issue.  There 
were  two  criteria  for  these  decisions.  They  /were:  1)  the 

relative  importance  of  the  issue  to  the  problem.;  and  2) 
the  availability  of  data  to  support  an  evaluation  of  the 
issue.  The' issues  treated  most  extensively  are  those  which 
were  considered  most  important  and  for  which  data  was  avail 
able  or  obtainable.  The  issues  treated  least  extensively, 
or  ignored,  are  those  which  were  considered  less  i.mportant 
and  for  which  no  data  was  available  or  reasonably  obtain- 
able . 


V.2.4  Additional  Data 

At  the  time  when  the  specific  issues  were  selected 
for  evaluation,  the  data  which  were  available  to  support 
the  evaluation  included  the  history  of  the  PEE  process, 
Thompson's  written  description  of  the  process,  and  the  writ 
ten  records  and  the  author's  recollections  of  ETIP's  use 
of  the  process  which  were  used  to  construct  the  chronology. 
The  process  of  selecting  issues  helped  to  highlight  those 
for  which  this  data  was  insufficient  to  enable  presentation 


of  conclusions  which  represented  anything  more  than  the 
author’s  own  beliefs.  Questionnaire  items  were  written 
for  those  issues  where  it  seemed  that  additional  information 
could  be  obtained  from  other  participants  in  the  process 
which  would  enable  a more  confident  evaluation  of  specific 
issues . 

The  items  were  divided  into  a questionnaire  for  panel 
members  and  a questionnaire  for  representatives  of  bidders’ 
firms.  Each  questionnaire  was  then  pilot  tested.  The  panel 
questionnaire  was  given  to  a person  who  was  familiar  with 
PEE  through  his  participation  on  later  panels  where  ETIP 
used  the  process.  The  bidder  questionnaire  was  given  to 
two  people  in  a firm  which  had  bid  on  subsequent  source 
selections  where  ETIP  used  the  process.  Some  minor  modifica- 
tions were  made  in  the  wording  of  several  items  and  addi- 
tional background  material  was  included  with  the  question- 
naire as  a result  of  the  pilot  test.  The  test  also  indi- 
cated that  response  to  the  bidder  questionnaire  is  sensitive 
to  the  person  in  the  firm  who  answers  the  items.  The  two 
individuals  in  the  same  firm  who  were  both  closely  involved 

in  preparing  their  firm’s  proposal  had  divergent  responses 

1 

to  some  of  the  items. 

1 

This  served  to  highlight  the  importance  of  knowing 
the  values  and  beliefs  of  individuals  and  what  role  they 
will  have  in  management  of  the  system  development.  Firms 
are  not  monolithic  in  the  way  they  view  a given  problem 
and  thereby  in  the  way  they  manage  an  approach  to  solving 
it . 


The  method  for  administering  the  items  combined  a ques- 

2 

tionnaire  and  interview  format.  Questionnaires  were  mailed 
to  participants  together  with  some  background  information 
on  the  source  selection  (a  list  of  the  factors  and  sub- 
factors and  a list  of  other  material  which  had  been  included 
in  the  RFP ) to  help  remind  them  of  the  process.  The  bidder 
questionnaire  was  mailed  to  the  person  in  the  firm  who  ap- 
peared to  have  managed  the  firm's  response  to  the  RF?. 

The  cover  letter  indicated  that  the  author  would  phone  to 

•3 

provide  clarifications  and  to  obtain  their  responses.*' 
Participants  were  then  interviewed  after  having  at  least 
a week  to  review  the  items. 

During  the  interviews  each  item  was  reviewed  with  the 
respondents.  For  the  yes/no  answers,  people  frequently 
provided  additional  information  to  justify  or  qualify  their 
responses.  After  all  of  the  questionnaire  items  were  an- 
swered, the  author  asked  two  additional  questions.  One 
was  whether  it  was  difficult  to  recall  the  events  and  their 
reactions  related  to  the  items  they  were  being  asked  about. 
The  other  was  whether  there  was  anything  which  they  had 
not  already  been  asked  about  which  stood  out  in  their  mind 

2 

Several  were  hand  delivered. 

3 

One  participant,  who  was  called  in  advance  to  obtain 
his  new  address,  asked  that  additional  background  informa- 
tion be  sent  with  the  questionnaire.  In  this  case,  the 
set  of  illustrative  questions  was  also  included  with  the 
sub-f  actors . 


concerning  the  source  selection  process.  Copies  of  the 
questionnaires  and  cover  letters  are  presented  in  the  Ap- 
pendix. 

Response  to  the  questionnaires  was  very  good.  All 
panel  members  were  interviewed  and  all  but  one  bidder  repre- 
sentative were  interviewed.  The  one  bidder  was  traveling 
quite  extensively  during  the  three  weeks  the  author  tried 
to  speak  with  him  and  could  not  be  reached. 

Of  the  seventeen  panel  members,  fifteen  provided  full 
responses  and  two  provided  partial  responses.  The  two  who 
provided  partial  responses  indicated  that: 

o I didn’t  remember  participating  and  would  be  giving 
reactions  to  your  current  description  of  the  pro- 
cess ; 

o I remember  that  it  was  an  interesting  day  and  that 
the  proposers  that  came  to  the  top  we  all  agreed 
and  they  were  indeed  the  best. 

The  first  had  participated  on  four  sub-factors  and  did  not 
answer  any  of  the  questionnaire  items.  The  second  had  par- 
ticipated on  two  sub-factors  and  only  answered  several  of 
the  questionnaire  items.  All  six  out  of  the  seven  repre- 
sentatives of  bidders  firms  who  were  interviewed  provided 
full  responses. 

V.2.5  General  Parameters  to  the  Evaluation 

When  interpreting  the  data  to  determine  the  effective- 
ness of  PFE  in  dealing  with  the  specific  issues,  it  is  impor- 

4 

One  panel  member  mailed  his  questionnaire  back  and 
was  therefore  not  asked  these  questions. 


tant  to  consider  parameters  which  might  lead  to  plausible 
rival  interpretations.  A summary  list  of  general  parameters 
is  presented  in  Figure  V.2  and  discussed  in  the  next  several 
paragraphs . 


o Memory 

0 Subsequent  Experience  with  the  Systems  Development 
o Subsequent  Experience  with  the  PFE  Process 
o Other  Related  Source  Selection  Experience 
o Outcome  of  the  Source  Selection 

o Ego  Issues  Related  to  Arguments  About  What  To  Do 
o Lying  by  Respondents  or  the  Author 

FIGURE  V.2  GENERAL  PARAMETERS  TO  THE  EVALUATION  OF  THE 
OUTCOME  OF  ETIP’s  USE  OF  PFE 


V.2. 5.1  Memory . One  of  the  main  parameters  is  memory. 
Much  of  the  data  used  in  this  case  study  is  it  is  based 
on  the  recollections  of  various  participants  in  the  source 
selections  and  was  collected  three  to  four  years  after  the 
events  occurred.  Present  recollections  of  facts  and  prior 
opinions  may  contain  errors  due  to  difficulty  of  recall. 

Respondents  were  asked  how  much  of  a problem  recall 
was,  in  order  to  obtain  their  self-report  on  this  parameter. 
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Of  the  fourteen  panel  members  who  responded  to  this  item, 
three  said  that  memory  was  a problem,  seven  said  that  it 
was  not,  and  four  said  that  it  presented  some  problems. 

Of  the  six  bidders,  one  said  that  memory  was  a problem, 
four  said  that  it  was  not,  and  one  said  that  it  presented 
some  problems.  Figures  V.3  and  V.4  list  additional  responses 
under  each  of  these  categories. 

It  should  be  noted  that  a response  of  ”no"  does  not 
necessarily  mean  that  a respondents*  recall  was  always  ac- 
curate. The  author  found  factual  errors  in  the  recall  of 
at  least  one  panel  member  who  reported  that  memory  was  not 
a problem. 

V.2.5.2  Subsequent  experience  with  the  systems  develop- 
ment . Many  of  the  participants  in  the  source  selection 
subsequently  had  a role  in  working  on  the  development  of 
the  evaluation  systems.  This  is  true  for  some  of  the  panel 
members,  the  contract  negotiator,  the  author  and,  of  course, 
the  successful  bidders.  Responses  by  these  people  concern- 
ing the  effectiveness  of  the  PFE  process  in  dealing  with 
some  of  the  issues  of  source  selection  may  be  mediated  by 
these  later  experiences.  No  data  was  collected  to  try  to 
measure  participants  on  this , parameter . 

V.2.5.3  Subsequent  experience  with  the  PFE  process. 

ETIP  has  used  the  PFE  process  on  two  subsequent  procurements. 
One  each  in  1977  and  1978.  Some  of  the  participants  in 
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Was  recall  a problem  in  responding  to  the  questionnaire 

items? 

Yes  - 3 No  - 7 Somewhat  - 4 

Remarks  by  those  answering  Yes 

o Remembering  was  a problem,  particularly  memoriesof  spe- 
cific exchanges,  circumstances,  and  my  own  reactions  (PM5). 

o Memory  was  quite  a problem.  The  reprint  of  the  factors 
and  sub-factors  helped  (PM9). 

o Without  seeing  a proposal  it  was  very  hard  to  remember 
(PM4) . 

Remarks  by  those  answering  No 

0 No,  but  these  were  reflex  answers.  I might  have  answered 
differently  if  you  asked  closer  to  the  events  (PM11). 

o No,  since  I had  some  notes  from  the  experience  to  help. 
Otherwise  it  would  have  been  more  difficult  (PM1). 

o No,  it  stuck  in  my  mind  because  it  was  unusual  (PM15). 

o Not  a major  one.  It  made  quite  an  impression  on  me. 

It  was  the  most  elaborate,  largest  exercise  of  this  type 
I'd  ever  been  through.  There  may  have  been  a lot  of  spe- 
cific points  I would  have  remembered  if  it  weren't  four 
years  ago.  What  stands  out  are  the  elaborateness  of  the 
process,  the  fact  that  I didn't  find  that  to  be  a problem, 
the  interaction  among  the  people  who  reviewed  each  section 
was  very  constructive,  and  the  overwhelming  sense  that 
it  was  an  inappropriately  written  RFP  (PM3). 

Remarks  by  those  answering  Somewhat 

0 My  answers  were  more  on  impressions  than  on  substance. 

I do  not  remember  any  of  the  substance  of  the  proposals 
(PM10). 

o My  overall  impressions  of  strengths  and  weaknesses  are 
accurate.  Some  things  were  a little  hard  to  remember 
(PM6) . 

FIGURE  V.3  PANEL  MEMBERS'  SELF-REPORTS  ON  PARAMETER  OF  RECALL 


o I remembered  fairly  well,  even  specific  deliberation. 

It  wasn’t  easy  though.  I have  a moderate  amount  of  memory 
of  it  (PM7). 

0 It  wasn’t  as  bad  as  I thought  it  was  going  to  be,  but 
it  was  still  somewhat  of  a problem.  Since  I haven’t  been 
on  other  panels  it  made  it  easier  (PM16). 

FIGURE  V.3  Continued 


Was  recall  a problem  in  responding  to  the  questionnaire 

items? 

Yes  - 1 No  - 4 Somewhat  - 1 

Remarks  by  person  answering  Yes 

0 As  I thought  about  it  more  items  of  memory  seemed  to  be 
coming  back  to  me  (BR1). 

Remarks  by  those  answering  No 

0 If  you  had  not  sent  me  the  material  I would  not  have  felt 
as  comfortable  with  my  answers  (BR4). 

0 Not  in  relation  to  the  questions  you  posed,  given  the 
appendix  you  attached  (BR6). 

o Not  the  least  bit.  It  was  a big  job  to  propose  on  it. 

I didn’t  even  have  to  refer  to  the  extracts  which  you 
included  (BR3). 

o I’ll  never  forget  that  RFP.  It  was  quite  different  (BR2). 

Remarks  by  person  answering  Somewhat 

o Not  my  feelings  about  the  process.  What  we  wrote,  I don’t 
remember  (BR5). 

FIGURE  V.4  BIDDER  REPRESENTATIVES’  SELF-REPORTS  ON 
PARAMETER  OF  RECALL 


the  1976  source  selection  participated  in  one  or  both  of 
these  later  procurements.  The  author,  for  example,  was 
closely  involved  in  writing  project  plans,  writing  portions 
of  the  RFPs,  and  coordinating  the  evaluation  panel  meetings 
for  both  of  the  later  procurements.  The  same  contract  nego 
tiator  was  responsible  for  the  1977  procurement.  One  of 
the  bidders  on  the  1976  source  selection  also  bid  in  1977. 
Some  of  the  same  panel  members  took  part  in  one  or  both 
of  the  later  procurements.  The  responses  by  these  people 
concerning  the  1976  source  selection  may  be  influenced  by 
their  experiences  in  these  later  procurements.  No  special 
data  was  collected  to  try  to  measure  the  effect  of  this 
parameter  on  responses. 

V.2.5.4  Other  related  source  selection  experience. 
Experiences  of  participants  in  system  acquisition,  and  spe- 
cifically source  selection,  probably  effects  their  reaction 
to  the  PEE  process  and  therefore  their  answers  to  the  ques- 
tionnaire items.  If  a respondent  has  no  system.s  or  related 
experience,  they  may  not  understand  some  of  the  problems 
which  the  process  was  trying  to  address.  Two  items  were 
included  in  the  questionnaires  in  order  to  obtain  an  indica 
tion  of  experience  in  source  selection.  One  asked  how  many 
RFPs/evaluat ion  panels  the  respondent  had  responded  to/serv 
on.  The  other  asked  how  many  of  these  were  for  projects 
of  similar  size,  complexity  and  uncertainty  to  this  one. 
Definitions  of  complexity  and  uncertainty  were  given  in 
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the  background  note  at  the  beginning  of  the  questionnaire. 
These  terms  were  used  to  describe  characteristics  of  systems 
related  experience.  It  is  possible  that  respondents  inter- 
preted them  differently  than  the  author  intended.  The  re- 
sponses to  these  items  are  presented  in  Figures  V.5  and 
V.6.  As  might  be  expected  there  is  quite  a bit  of  variation 
in  experience  among  respondents,  both  panel  and  bidder. 

V.2.5.5  Outcome  of  the  Source  Selection.  * Another 
parameter  to  consider  is  the  effects  of  the  outcome  of  the 
source  selection.  Responses  to  subsequent  questions  about 
the  effectiveness  of  PFE  in  solving  specific  problems  may 
be  partially  a function  of  who  won.  A bidder  who  lost  may 
find  the  process  less  effective  than  one  who  won.  A panel 
member  who  disagreed  with  the  selection  of  the  successful 
bidder  may  be  biased  against  the  process  and  visa  versa. 

No  specific  measurement  was  attempted  to  try  determine  the 
extent  to  which  this  affected  responses. 

V.2.5.6  Ego  issues  related  to  arguments  about  what 
to  do . Chapter  IV  indicated  that  there  was  heated  debate 
among  some  of  the  panel  members  concerning  many  of  the  events 
related  to  the  procurement.  Some  involved  the  selection 
of  PFE  as  the  method  which  would  be  used  for  the  source 
selection.  There  were  also  many  clashes,  after  the  signing 
of  the  contracts,  concerning  the  development  of  the  systems. 
It  is  possible  that  the  responses  of  some  of  the  panel  mem- 
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Approximately  how  many  other 
evaluation  panels  have  you 
served  on? 

Of  the  total  number  of 
other  proposal  evaluation 
panels  which  you  have  served 
on,  about  how  many  were 
for  projects  of  the  m.agni- 
tude  (i.e.,  200  man-months), 
complexity,  and  uncertainty 
of  this  one? 

1 /O 

(PM10 ) 

3-4/3-^ 

(PM1 1 ) 

10-11/3-4 

(PM12) 

20/8 

(PM1  ) 

10/7 

(PM13) 

35/5 

(PM2) 

30/3-4 

(PM50) 

10/0 

(PM14) 

6/3 

(PM9  ) 

0/0 

(PM6) 

25/5 

(PM7) 

2/2 

(PM8) 

6-7/1 

(PM4) 

3/0 

(PM15) 

6/0 

(PM3) 

0/0 

(PM16 ) 

FIGURE  V.5  PANEL  MEMBERS' 


SOURCE  SELECTION  EXPERIENCE 


Approximately 
RFPs  have  you 

how  many  other 
responded  to? 

Of  the  total  number  of 
RFPs  which  you  have  re- 
sponded to,  about  how  many 
were  for  projects  of  simi- 
lar size  (i.e.,  200  man- 
months),  complexity,  and 
uncertainty  as  this  one? 

150/20 

(BR5) 

600-1000/175 

(BR4) 

100/25-50 

(BR6) 

100/less  than  12 

(BR3) 

30/2-3 

(BR2) 

30/5 

(BR1  ) 

FIGURE  V.6  BIDDER  REPRESENTATIVES’  SOURCE  SELECTION 
EXPERIENCE 


bers  may  have  been  influenced  by  ego  related  issues  such 
as  the  degree  to  which  they  felt  involved  in  the  development 
of  the  RFP  and  the  degree  to  which  they  perceived  they  had 
influence  in  various  parts  of  the  systems  developments. 
Though  no  specific  measurement  was  attempted  here,  it  is 
unlikely  that  this  parameter  affected  the  responses  of  more 
than  two  or  three  panel  members. 

V.2.5.7  Lying.  A final  parameter  to  consider  here 
is  whether  the  respondents  are  telling  the  truth.  For  re- 
sponses from  bidder  representatives  this  may  be  mostly  a 
function  of  whether  the  person  who  knows  the  truth  was  the 


In  a large  RFP  many  dif- 


one  who  answered  the  questions, 
ferent  people  are  involved  in  preparation  of  the  proposal. 
The  answer  to  a question  is  partially  a function  of  whether 
the  author  located  the  person  who  was  responsible  for  the 
aspects  of  the  proposal  related  to  the  issue  under  question. 
For  the  bidders,  telling  the  truth  may  also  be  a function 
of  their  expected  future  with  ETIP.  If  they  feel  that  their 
responses  may  affect  this,  they  mav  portray  themselves  as 
more  favorable  than  they  actually  are.  For  the  panel  mem- 
bers (including  the  author)  and  for  the  contract  negotiator, 
there  is  the  possibility  that  the  truth  is  being  altered 
by  the  desire  to  not  admit  that  anything  might  have  been 
done  incorrectly.  This  is  part  of  a well  known  phenomenon 
in  government  circles  which  is  known  as  "covering  your  ass." 
The  responses  and  interpretations  of  the  author  may  also 
be  suspect  on  the  grounds  that  he  would  like  to  complete 
his  degree  requirements.  Since  his  committee  chairman  is 
the  primary  developer  of  PFE,  his  ability  to  be  candid  may 
be  questioned  by  some  readers.  Responses  of  at  least  two 
of  the  panel  members,  the  COTR,  and  Thompson  might  also 
be  questioned  on  the  basis  that  they  have  a stake  in  the 
success  of  PFE  and  are  favorably  biased  toward  it. 

The  author’s  impression  was  that  all  of  the  bidder 
representatives  who  were  interviewed  were  closely  involved 
in  the  preparation  of  their  firm’s  proposal  in  response 
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to  the  RFP.  It  also  seemed  that  the  respondents  were  quite 
I candid  in  their  responses  to  the  questionnaire  items.  How- 

ever, as  one  respondent  was  careful  to  point  out,  the  an- 
swers represented  his  personal  views  and  not  necessarily 
j those  of  the  firm.  Others  who  worked  on  the  proposal  may 

have  had  quite  different  responses. 

[ The  author  believes  that  the  panel  members  were  equally 

candid  in  their  responses.  In  part  this  may  be  due  to  the 
fact  that  most  of  these  people  no  longer  have  any  relation- 
ship with  the  resulting  projects  or  with  ETIP.  Some  who 
were  outside  evaluators  never  had  a close  relationship  with 
the  project  and  others  who  were  ETIP  or  PAA  employees  have 
since  taken  new  jobs.  No  special  steps  have  been  taken 
to  control  for  the  responses  of  the  author,  the  COTR,  or 
Thompson.  However,  steps  have  been  taken  to  try  to  compen- 
sate for  biases  which  may  affect  the  author’s  interpreta- 
tions of  data.  An  attempt  has  been  made  to  provide  the 
reader  with  as  much  unfiltered  information  describing  the 
process  and  data  from  the  process  as  possible.  This  has 
I been  done  through  including: 

o the  history  of  the  process  (Section  II. 4) 

1 o the  chronology  of  ETIP’s  use  of  the  process 

(Chapter  IV), 

) o prior  documents  from  and  about  the  RFP  (Appen- 

dix), 

1 

I 


o respondents'  elaborations  on  their  responses 
to  the  questionnaire  items  (Figures  in  this 
chapter ) , and 

o the  actual  ratings  from  the  evaluation  process 
(Appendix ) . 

It  is  hoped  that  this  information  will  enable  the  read- 
ers to  form  their  own  evaluations  on  both  the  issues  treated 
extensively  by  the  author  and  on  those  issues  which  readers 
feel  are  important  but  which  the  author  has  selected  to 
treat  in  less  depth. 

V . 3 Evaluation  of  the  Parametric  Factor  Evaluation  Approach 

Against  the  Specific  Critical  Issues 

V.3.1  Introduction 

This  section  is  divided  into  sub-sections  according 
to  the  category  groupings  of  the  issues.  For  each  sub- 
section the  category  is  described  and  the  issues  which  are 
evaluated  within  it  are  listed. 

The  paragraphs  in  each  sub-section  which  follow  the 
category  description  are  used  for  the  evaluation  of  the 
issues  within  that  category.  Each  issue  is  identified  and 
its  significance  is  noted.  A statement  is  then  made  about 
whether  the  information  available  indicates  that  ETIP  was 
successful  or  unsuccessful  at  dealing  with  the  issue  by 
using  PFE  or  whether  the  data  are  equivocal.  The  informa- 
tion used  in  reaching  this  conclusion  is  then  described. 

In  some  cases  this  information  will  be  from  the  analysis 


of  questionnaires  sent  to  participants  in  the  process  or 
analysis  of  the  factor  teams’  ratings.  In  other  cases  the 
author  merely  reports  the  results  as  he  was  able  to  observe 
them  through  his  participation  in  the  process.  An  identify 
ing  number  was  assigned  to  each  of  the  respondents  so  that 
readers  may  compare  their  remarks  across  items  when  review- 
ing the  Figures  in  this  section. 

V.3.2  Effective  Communication 

The  category  of  effective  communication  encompasses 
issues  dealing  with  both  format  and  content  requirements 
of  the  RFP  and  the  proposals.  The  issues  are  concerned 
with  whether  PFE  provided  for  effective  two  way  communica- 
tions between  the  government  and  potential  bidders  during 
the  source  selection.  That  is,  whether  the  information 
provided  by  the  government  and  the  form  in  which  it  was 
provided  allowed  bidders  to  assess  what  needed  to  be  done 
and  whether  the  responses  and  the  form  in  which  they  were 
required  enabled  the  evaluation  panel  to  accurately  deter- 
mine the  capabilities  of  the  bidders. 

Eight  issue  discussions  are  included  under  this  cate- 
gory. Four  are  primarily  related  to  format  of  communica- 
tions and  three  are  primarily  related  to  content  of  communi 
cations.  The  issues  are: 


Format 


i?e 


1. a)  Did  PFE  provide  too  much  structure  for  the 

bidders'  response? 

Is  the  bidder  able  to  display  his  compe- 
tence? 

Does  it  present  problems  for  the  bidder 
or  panel  when  no  formal  structure  is  used 
for  the  object  of  the  procurement? 

b)  Did  the  panel  have  difficulty  evaluating 
the  proposals  on  a sub-factor  basis? 

Was  the  panel  able  to  find  the  information 
it  needed? 

Did  they  feel  that  the  evaluation  was  being 
done  out  of  context  from  the  project? 

2.  Should  the  size  of  the  proposals  have  been 
limited? 

Did  length  correlate  with  final  score? 

3.  Did  the  bidders  understand  what  the  evalua- 
tion panel  looked  at  in  making  their  evalua- 
tions? 

Did  knowing  this  help? 

4.  Did  the  process  enable  the  panel  to  deal 
effectively  with  differences  in  information 
among  bidders  during  the  evaluation? 

Did  the  question  form  provide  an  effective 
mechanism  for  obtaining  missing  information 
and  clarification  after  the  proposals  were 
received? 

Content 

l.a)  Did  the  process  communicate  to  bidders  what 
it  was  the  government  wanted  to  buy? 

Did  bidders  know  what  they  were  bidding 
on? 

b)  Did  the  process  communicate  to  prospective 
bidders  the  skills  which  would  be  required 
to  do  the  work  on  the  project? 

Were  many  of  the  sub-factors  not  useful  to 
the  bidders  or  the  panel  (e.g.,  the  Manage- 
ment or  Reports  & Reviews  sub-factors)? 


2. 


3.  Was  parroting  back  of  information  from  the 
RFP  by  bidders  to  the  exclusion  of  other 
information  a problem  for  the  panel? 

V.3.2.1  Effectiveness  of  the  structure.  The  first 
format  issue  is  whether  PFE  provided  too  much  structure 
for  the  bidder’s  response.  The  primary  concern  here  was 
that  constraining  bidders  to  follow  the  sub-factor  structure 
would  lessen  their  ability  to  effectively  demonstrate  their 
qualifications.  The  corollary  of  this  concern  was  that 
the  sub-factor  structure  would  interfere  with  the  panel's 
ability  to  evaluate  effectively  the  capability  of  the  bid- 
ders. The  aspect  of  PFE  which  is  salient  to  this  issue 
is  that  it  did  not  provide  for  information  exchange  in  terms 
of  the  object  or  program  to  be  procured.  Bidders  were  not 
asked  to  propose  a design  for  the  evaluation  systems.  They 
were  asked  instead  to  respond  to  the  problems  which  the 
government  thought  would  arise  in  developing  such  a system. 
At  least  one  panel  member  voiced  concern  during  a team  meet- 
ing that  this  reduced  the  bidders’  ability  to  demonstrate 
their  capabilities. 

The  structure  was  generally  well  received  by  both  the 
panel  and  the  bidders.  A question  with  three  sub-items 
was  included  in  both  the  panel  and  the  bidder  questionnaires 
to  obtain  reactions  to  the  issues  related  to  the  structure. 
The  items  and  responses  to  them  are  presented  in  Figures 
V.7  to  V.12.  Figure  V.13  lists  responses  to  the  open  ended 


questionnaire  items  which  pertain  to  structure.  The  panel 

members  generally  felt  that: 

o the  structure  enabled  bidders  to  demonstrate 
their  ability  to  deal  with  the  critical  prob- 
lems of  the  project  (Yes  - 13;  No  - 1 ; Yes 
& No  - 1 ) ; 

0 the  structure  did  not  create  any  critical  gaps 
in  evaluating  bidders'  capabilities  (Yes  it 
created  gaps  - 4;  No,  it  did  not  create  gaps 
- 10;  Don't  know  - 1 K’  and 

0 information  relevant  to  the  issues  being  evalu- 
ated was  readily  located  in  the  proposals  (Yes 
-14 ; No  - 1 ) . 

One  aspect  of  structure  which  came  out  in  the  response 
of  some  panel  members  is  the  tradeoff  between  consistency 
of  evaluation  across  proposals  which  occurs  when  using  the 
PFE  structure,  and  the  possibility  of  missing  some  insights 
which  might  have  been  presented  by  bidders  under  a less 
structured  format.  In  fact  one  panel  member  felt  that  the 
structure  was  both  the  strongest  and  the  weakest  aspect 
of  the  process  (see  Figure  V.13).  The  author  believes  that 
this  tradeoff  needs  to  be  examined  as  a function  of  what 
the  government  is  trying  to  procure.  If  the  object  of  the 
procurement  is  uncertain  and  complex,  a structure  which 
tends  to  be  free  of  gaps  and  consistent  across  proposals 
is  probably  more  appropriate  than  one  which  evaluates  pro- 
posals for  solutions  which  may  be  inappropriate  under  a 
changed  environment. 


Did  you  feel  that  the  factor/sub-factor  structure  enabled 
bidders  to  demonstrate  their  ability  to  deal  with  critical 
problems  which  might  arise  during  the  project? 

Yes  - 13  No  - 1 Yes  & No  - 1 

Remarks  by  those  answering  Yes 

None 

Remark  by  person  answering  No 

o It  confined  bidders  to  a tight  recipe  for  how  to  answer 
and  did  not  provide  them  with  the  ability  to  innovate. 

It  didn't  demonstrate  their  ability  to  deal  with  the 
critical  problems.  You  only  found  out  which  contractor 
could  write  best.  They  could  demonstrate  that  they 
could  identify  problems  but  I don't  think  they  could 
show  whether  they  could  deal  with  the  problem  (PM4) 

Remark  by  person  answering  Yes  and  No 

o Yes,  the  structure  covered  it  and  allowed  bidders  to  demon 
strate  their  ability. 

No.  it  got  too  refined.  Too  many  sub-factors.  The  kind 
of  interconnection  between  judgments  gets  lost.  There 
were  too  many  factors  and  sub-factors  by  a factor  of  about 
two.  The  result  of  the  judgment  was  not  necessarily  the 
sum  of  the  parts  (PM9) 

FIGURE  V.7  PANEL  MEMBERS'  RESPONSES  TO  QUESTION  la 
ON  THE  EFFECTIVENESS  OF  THE  STRUCTURE 


Did  you  feel  that  the  factor/sub-factor  structure  enabled 
bidders  to  demonstrate  their  ability  to  deal  with  the  prob- 
lems which  might  arise  during  the  project? 

Yes  - 4 No  - 2 

Remarks  by  those  answering  Yes 

0,1  thought  it  was  too  structured  (BR2). 

0 It  was  fairly  well  thought  out.  It  covered  all  of  the 
points  quite  well,  to  the  point  of  exhaustion.  There 
seemed  to  be  some  repetition  (BR1). 

Remarks  by  those  answering  No 


o I think  the  structure  stunk.  The  approach  is  suitable 
for  hardware  not  program  evaluation  and/or  experimental 


design.  When  dealing  with  program  evaluation  you  are 
dealing  with  nominal  data  at  best  and  therefore  the  summa- 
tion of  nominal  data  might  not  hold.  It  also  forced  peopl 
to  prepare  proposals  in  a non-research  way.  One  that 
would  be  used  for  an  RFQ,  not  an  RFP.  The  project  was 
poorly  defined  and  so  were  the  parts  of  the  proposal. 

The  documents  loosely  pointed  to  the  evaluation  but  there 
wasn’t  a clear  set  of  program  or  project  objectives  (BR5). 

o We  were  trying  to  create  objectivity  where  the  problems 
arising  and  the  sensitivities  that  had  to  be  appealed 
to  were  things  which  were  subjective.  The  desire  to  be 
objective  is  to  be  applauded.  However,  we  tried  to  create 
objectivity  where  it  didn’t  exist.  That’s  not  saying 
it  wasn’t  a good  RFP  (BR4). 

FIGURE  V.8  BIDDER  REPRESENTATIVES'  RESPONSES  TO  QUESTION  la 
ON  THE  EFFECTIVENESS  OF  THE  STRUCTURE 


Did  you  feel  that  there  were  critical  gaps  in  evaluating 
the  bidders’  capabilities  to  perform  effectively  on  the 
project? 

Yes  - 4 No  - 10  Don’t  know  - 1 

Remarks  by  those  answering  Yes 

o It’s  strictly  a feeling  only.  I don’t  remember  what  they 
were.  I was  somewhat  uneasy  about  how  well  I understood 
it  ( PM1 6 ) . 

o Some  of  the  sub-factor  statements  seemed  too  theoretical. 
I’m  not  sure  it’s  a result  of  the  structure.  For  example, 
sub-factor  1.2.  You  can’t  measure  that  sub-factor  as 
it  is  stated.  It  needs  to  be  broken  down  further  to  make 
it  operational  (PM1). 

o The  structure  of  the  factors  and  sub-factors  was  too  com- 
plicated. Some  elements  of  the  required  skills  were  in- 
cluded in  more  than  one  factor,  or  at  least  the  possibil- 
ity of  this  seemed  to  exist.  The  large  number  of  sub- 
factors meant  the  distribution  of  points  was  too  great 
to  enable  sufficient  weighting  of  the  most  important. 
Although  this  was  not  necessarily  the  case,  it  had  a ten- 
dency to  work  out  that  way  (PM2). 

0 The  evaluation  criteria  and  the  RFP  were  written  from 
an  academic,  formal,  research  design  perspective  that 
rewarded  firms  who  could  respond  as  if  this  were  an  exam- 
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ination  in  graduate  school,  rather  than  an  opportunity 
to  develop  a process  for  program  evaluation  in  the  real 
world.  Thus,  a bidder’s  recognition  of  the  constraints 
of  reality  of  evaluation  was  not  given  much,  if  any,  weight, 
even  if  such  considerations  were  present  in  the  proposal. 

The  RFP  discouraged  bidders  from  including  such  discus- 
sions (PM3). 

Remark  by  a person  answering  No 

o I don’t  remember  how  the  proposals  were  written  so  I don’t 
know  if  there  were  any  gaps.  I guess  there  probably  weren’t 
any  critical  gaps  (PM4). 

Remark  by  person  answering  Don’t  know 

o Any  structure  presents  problems  of  addressing  questions 
at  interfaces  between  sets  of  factors/sub-factors.  I 
don’t  remember  whether  I sensed  that  with  this  evaluation. 
What  most  of  us  do  in  evaluating  complicated  proposals 
is  form  an  overall  impression  and  then  segment  it.  This 
process  assured  that  we  all  segmented  it  in  the  same  way. 

It  had  the  advantage  of  consistency  and  the  disadvantage 
that  if  we  had  done  it  differently  we  might  have  received 
some  added  insight  (PM5). 

FIGURE  V.9  PANEL  MEMBERS’  'RESPONSES  TO  QUESTION  1b  ON  THE 
EFFECTIVENESS  OF  THE  STRUCTURE 


Did  following  the  factor/sub-factor  structure  create  any 
problems  for  your  team,  in  trying  to  demonstrate  its  cap- 
ability, which  would  not  have  been  present  if  a different 
structure  had  been  used? 

Yes  - 4 No  - 2 

Remarks  by  those  answering  Yes 

o It  was  a problem  of  trying  to  prepare  a proposal  to  con- 
form to  an  evaluation  structure  rather  than  the  design 
of  the  project.  It  was  designed  to  facilitate  the  giving 
of  scores  to  be  added  up  and  although  I can  understand 
the  desire  to  do  that,  from  the  reviewers’  point  of  view, 
I don’t  see  that  as  the  way  to  present  an  approach  to 
a problem.  An  alternative  would  be  to  post  a sample  task 
and  evaluate  everyone  on  their  response  to  it.  People 
would  be  bidding  on  the  same  job  or  at  least  the  same 
task  (BR5). 

0 It  was  a new  structure.  The  biggest  problem  was  figuring 
out  what  you  meant.  You  came  out  with  a glossary.  The 


definitions  were  not  clear  enough.  They  were  generally 
too  short.  On  sub-factor  1.2  (Importance  of  Program  to 
Bidder),  you  can’t  do  this  with  a large  corporation.^ 

If  it’s  a large  corporation,  it’s  going  to  do  it.  If^ 
it’s  a new  corporation,  it’s  going  to  go  beyond  what  it^ 
said  in  the  proposal.  The  point  is  that  you  really  can't 
measure  that.  It  is  subjective,  not  objective  (3RU). 

o The  cost  to  develop  it  was  rather  exorbitant  for  a small 
firm.  Following  the  structure  was  not  difficult.  It 
didn’t  create  problems^to  understand  it.  The  relationship 
of  the  factors  and  sub-factors  was  excellent  (ER3). 

o There  was  repetition  which  made  it  very  tedious  and  dis- 
couraged people.  It  was  too  highly  structured  with  ap- 
parent repetition  of  things  (BR2). 

Remarks  by  a person  answering  No 

o It  forced  us  to  go  through  each  of  the  points  you  wanted 
us  to  go  through.  The  redundancy  may  have  inhibited  our 
style  of  writing.  On  some  of  the  points  we  felt  that 
we  didn’t  have  much  to  say  or  that  they  didn’t  deserve 
to  have  much  said  about  them,  but  that  may  have  been  an 
indication  of  some  areas  in  which  we  were  lacking  (3R1). 

FIGURE  V.10  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
lb  ON  THE  EFFECTIVENESS  OF  THE  STRUCTURE 


Was  your  team  in  any  manner  placed  at  a competitive  disadvan- 
tage because  you  had  to  follow  the  factor /sub-factor  struc- 
ture? 

Yes  - 2 No  - 3 Don’t  know  - 1 

Remarks  by  those  answering  Yes 

o It  inhibited  market  entry  for  smaller  firms  to  bid  on 
a larger  project,  because  of  the  manpower  required  (3R5). 

o Because  we  are  small  and  we  try  to  put  our  best  foot  for- 
ward. Small  companies  are  not  as  able  to  bear  the  cost. 

It  took  a hell  of  a lot  of  work,  research,  and  coordina- 
t ion  (BR3 ) . 

Remarks  by  those  answering  No 

o Because  everyone  had  to  go  through  it.  It  created  an 
element  of  confusion  and  unnecessary  concern  because  of 
its  newness  (BR4 ) . 


Remark  by  person  answering  Don’t  know 


0 Those  people  who  evaluated  the  submittals  would  be  able 
to  tell  better.  At  the  time  I don’t  think  we  had  any 
feelings  of  being  placed  at  a disadvantage  (BR1). 

FIGURE  V.11  BIDDER  REPRESENTATIVES'  RESPONSES  TO  QUESTION 
1c  ON  THE  EFFECTIVENESS  OF  THE  STRUCTURE 


Were  you  able  to  readily  locate  the  information  in  the  pro- 
posals which  was  relevant  to  the  issues  you  were  evaluating? 

Yes  - 14  No  - 1 

Remarks  by  those  ansv/ering  Yes 

o In  most  cases.  Some  bidders  followed  the  directions  bet- 
ter than  others  (PM6). 

o There  may  have  been  one  or' two  exceptions.  It  was  a mat- 
ter of  the  bidder  not  following  directions  (PM7). 

o However,  sometimes  it  took  quite  a bit  of  reading  when 
there  was  extensive  cross-referencing  (PM8). 

o If  the  bidders  did  it  right  (PM4). 

Remark  by  person  answering  No 

o Some  of  the  bidders  didn’t  follow  directions.  In  cross- 
referencing  if  you  follow  it  to  where  they  referenced 
and  read  that  section,  they  still  haven’t  dealt  with  what 
they  should  have.  I don’t  think  the  problem  was  in  the 
instructions  in  the  RFP  (PM1). 

FIGURE  V.12  PANEL  MEMBERS’  RESPONSES  TO  QUESTION  1c  ON 
THE  EFFECTIVENESS  OF  THE  STRUCTURE 


Responses  From  Panel  Question  5 on  Strongest  Aspect  of  the 
Process 


o The  common  structure  for  the  evaluation.  It  was  a good 
structure.  Without  more  thought  or  experimentation  I 
can’t  say  whether  it  was  the  best  possible  structure  (PM5). 

o The  structure  provided  by  the  factors  simplified  the  pro- 
cess for  evaluating  and  comparing  proposals  (PM6). 


0 The  factors.  They  covered  many  of  the  essential  aspects 
and  having  them  specifically  laid  out  and  paid  attention 
to  was  the  great  strength  (PM9). 

Responses  From  Panel  Question  6 on  Weakest  Aspect  of  the 

Process 

0 The  common  structure  perhaps  inhibited  generation  of  in- 
sights which  might  have  contributed  something  of  value 
(PM5) . 

o It  was  a little  like  a college  exam.  You  lead  them  too 
far  along  so  you're  not  sure  how  much  is  their  writing 
what  they  think  would  be  a successful  response  vs.  their 
actual  capability  (PM10). 

Responses  From  Panel  Question  10  on  Any  Other  Aspects  of 

the  Process  Which  You  Wish  to  Get  Out 

o At  the  time  I had  an  instinctive  reaction  that  the  process 
was  overstructured.  I still  don't  know  whether  or  not 
it  was.  Defining  the  dimensions  and  defining  the  factors 
seems  hard  but  worthwhile  (PM5). 

o For  a million  dollar  or  more  proposal  you  would  need  some- 
thing like  this  to  be  able  to  find  things.  If  it  were 
$500,000  or  less  I wouldn't  bother  with  this  technique. 

It  would  be  overkill.  The  structured  approach  seemed 
better  for  guiding  the  reviewer  than  for  letting  the  pro- 
poser show  his  ingenuity.  Another  question  is  what  the 
process  did  to  keep  people  from  wanting  to  bid  (PM4). 

Responses  From  Bidder  Question  10  on  the  Strongest  Aspect 

of  the  Process 


o Concern  for  the  structural  process.  The  pre-bidders  con- 
ference. Giving  a lot  of  thought  to  weights  (3R2). 

o The  need  to  specify  ideas  and  expertise  in  all  of  the 
sub-groups  which  were  specified.  It  caused  us  to  test 
within  ourselves  whether  that  detail  met  the  objectives 
of  the  project  as  we  understood  it  (BR1). 

Responses  From  Bidder  Question  1 1 on  the  Weakest  Aspect 
of  the  Process 


o The  weakest  element  at  the  time  the  proposal  was  written 
was  the  confusion  of  the  new  system  which  we  had  to  re- 
spond to  (BR4). 

FIGURE  V.13  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  STRUCTURE 
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The  bidders  were  more  divided  in  their  reactions  to 
the  structure.  They  tended  to  feel  that  the  structure  en- 
abled them  to  demonstrate  their  ability  to  deal  with  the 
critical  problems  of  the  project  (Yes  - 4;  No  - 2).  They 
tended  to  feel  that  structure  created  some  problems  in  try- 
ing to  demonstrate  their  capability  which  would  not  have 
been  present  under  a different  structure  (Yes  - 4;  No  -2). 
They  were  divided  on  whether  they  felt  that  they  were  placed 
at  a competitive  disadvantage  by  the  structure, 

A remark  which  one  bidder  representative  (BR4)  made 
in  his  answers  to  the  questions  on  structure,  and  in  answers 
to  some  of  the  other  questions,  is  that  the  evaluation  tried 
to  measure  subjective  things  in  an  objective  manner.  He 
cites  sub-factor  1.2  (Importance  of  Program  to  Bidder)  as 
an  example,  saying  that  it  cannot  be  measured.  The  respon- 
dent seems  to  treat  objective  and  subjective  as  dichotomous 
variables.  The  author  believes  they  are  ends  of  a continuum. 
Nowhere  does  PFE  claim  that  the  measurements  of  bidders 
on  sub-factors  are  at  the  objective  extreme  of  the  continuum. 
The  method  of  measurement  is  for  the  panel  to  agree  on  sub- 
jective probability  estimates  through  sharing  all  available 
informationj  arguing  about  the  confidence  in  this  informa- 
tion, and  using  the  same  criteria  (understanding,  proposed 
approach,  and  present  or  potential  capability)  for  all  bid- 
ders. The  degree  of  subjectivity  may  vary  from  sub-factor 
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to  sub-factor  and  the  confidence  intervals  around  the  prob- 
ability estimates  on  each  sub-factor  may  vary  from  bidder 
to  bidder.  This  is  true  for  all  proposal  evaluation.  ?FE 
makes  confidence  in  the  ratings  explicit.  Where  the  confi- 
dence interval  is  wide  enough  so  that  a factor  team  feels 
that  a bidder  may  actually  be  unacceptable  on  a sub-factor 
where  his  score  is  acceptable  (and  vice  versa)  the  team 
notes  this  for  use  in  clarification  sessions. 

Under  PFE,  if  an  issue  is  considered  critical  to  the 
success  or  failure  of  the  project,  it  is  written  down  as 
a sub-factor  and  an  attempt  is  made  to  measure  it.  If  the 
issue  is  not  measureable,  then  it  cannot  be  used  as  a basis 
for  competitive  selection  (with  PFE  or  any  other  method). 

It  is  a risk  which  applied  across  all  bidders  and  the  pro- 
ject (Thompson  Note  1).  The  program  must  then  determine 
whether  it  is  willing  to  assume  this  risk  and  award  a con- 
tract. 


Sub-factor  1.2,  on  Importance  of  Program  to  Bidder, 
was  considered  to  be  one  of  the  most  critical  issues  of 
the  evaluation.  Because  of  the  uncertainties  invo.lved  in 
the  program,  including  changes  which  would  certainly  occur 
in  ETIP  and  PAA  management  and  staff,  it  was  important  that 
the  contractor  have  a continuing  long-term  interest  in  the 
implementation  of  the  systems.  This'  was  considered  to  re- 
quire evidence  of  an  interest  beyond  the  norm  of  the  bidder 


assuring  that  he  would  do  a good  job  because  the  firm  does 
a good  job  on  all  of  its  contracts.  Such  a commitment  may 
affect  issues  such  as  contractor  staff  stability,  access 
to  the  best  staff  in  the  firm,  and  the  willingness  of  the 
contractor  to  work  toward  the  long  term  requirements  of 
system  implementation,  rather  than  responding  to  short  term 
variations  in  government  objectives. 

This  issue  proved  troublesome  but  possible  to  measure. 
The  Management  Factor  Team  for  Agency  Impact  was  the  fourth 
team  to  meet.  After  deliberation  on  the  information  avail- 
able the  team  decided  that  it  was  unable  to  differentiate 
among  the  bidders  and  that  clarifications  would  be  required 
on  this  issue  with  any  bidder  who  appeared  to  be  possibly 
acceptable  overall.  All  bidders  were  rated  at  six.  As 
the  team  meetings  continued,  information  was  occasionally 
brought  out  which  related  to  sub-factor  1.2  (e.g.,  see  Sec- 
tion IV. 10.7,  paragraph  two).  This  information  was  avail- 
able for  use  in  the  oral  clarifications  and  for  use  in  rat- 
ing  firms  who  also  bid  on  the  commercial  impact  contract. 
Attention  was  paid  during  the  oral  clarifications  to  remarks 
which  related  to  the  firms’  commitments  to  developing  the 
systems.  When  all  available  information  was  considered, 

ETIP  was  able  to  differentiate  among  bidders  on  this  issue. 

5 

When  the  Management  factor  team  met  for  Commercial 
Impact  (the  6th  factor  team  on  that  contract)  it  was  able 
to  find  a basis  for  differentiating  among  the  bidders  on 
that  contract. 
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The  author  feels,  from  recalling  the  discussion  at  the  time, 
that  the  confidence  intervals  around  these  estimates  were 
greater  than  for  some  of  the  other  sub-factors,  but  that 
there  was  definitely  evidence  for  differentiation. 

Bidder  representatives'  responses  to  the  question  on 
whether  the  structure  created  any  special  problems  (Figure 
V.10)  did  not  show  any  patterns.  Four  respondents  indicated 
that  it  did  create  problems.  One  said  that  trying  to  under- 
stand what  ETIP  wanted  was  a problem.  Another  felt  that 
there  was  repetition  which  caused  problems.  A third  felt 
the  "relationship  of  the  factors  and  sub-factors  was  excel- 
lent 'but'  the  cost  to  develop  it  was  exorbitant."  The 
fourth  said  it  was  difficult  to  prepare  a proposal  to  conform 
to  an  evaluation  structure  but  didn't  specify  why. 

Two  bidders  felt  that  their  team  was  placed  at  a com- 
petitive disadvantage  by  the  structure  (Figure  V.11).  They 
felt  that  the  process  was  too  costly  for  small  firms  to 
respond  to.  This  is  clearly  an  issue  the  government  must 
consider  in  determining  whether  or  not  to  use  PFE.  The 
author  has  heard  estimates  of  cost  to  respond  placed  between 
$10,000  and  $20,000.  This  must  be  weighed  against  the  type 
of  project  and  the  total  estimated  cost  of  the  contract. 

If  a less  costly  process  will  enable  sufficient  confidence 
in  the  selection  of  a contractor,  given  the  nature  of  the 
contract,  then  PFE  should  not  be  used.  If  the  size  of  the 


contract  is  too  small  to  enable  bidders  to  invest  the  amount 
required  to  respond,  then  the  government  must  decide  whether 
other  available  procedures  will  provide  sufficient  confid- 
ence to  allow  the  government  to  assume  the  added  risks. 

A final  point  about  the  structure  of  PFE  is  its  strong 
interrelatedness.  Morris  (1973)  indicates  that  RFPs  are 
frequently  difficult  to  respond  to  because  the  various  parts 
do  not  mesh.  The  evaluation  requirements  do  not  correspond 
to  the  statement  of  work,  scope  of  effort,  etc.  In  PFE 
this  is  not  a problem.  The  statement  of  work,  scope  of 
effort,  schedule,  and  evaluation  criteria  are  all  tied  to 
the  factor/sub-factor  structure.  This  enables  a comparable 
basis  for  evaluating  capability,  division  of  effort,  and 
schedule . 

V.3.2.2  Size  of  proposals.  The  second  format  issue 
is  whether  a limitation  should  have  been  placed  on  the  size 
of  the  bidders’  proposals.  PFE  allows  the  proposals  to 
be  as  long  as  the  bidder  feels  is  necessary.  Some  members 
of  the  ETIP  and  PAA  staff  were  concerned  that  the  proposals 
which  would  be  received  would  be  excessively  lengthy  and 
would  take  an  unusually  long  time  to  evaluate  (see  section 
IV, 6).  This  is  related  to  one  component  of  what  the  litera- 
ture identifies  as  ’’goldplating”  of  the  proposal.  If  the 
bidder  feels  that  the  length  of  the  proposal  is  going  to 
be  one  of  the  major  factors  in  determining  their  rating, 
they  will  present  as  much  information  as  possible. 
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The  results  of  ETIP’s  source  selection  would  indicate 
that  there  was  no  need  to  place  a length  limitation  on  bid- 
ders' proposals.  The  proposals  did  not  seem  long  for  the 
size  and  nature  of  the  procurement.  The  time  to  evaluate 
them  did  not  seem  excessive. 

Proposals  ranged  from  90  pages  for  the  shortest  to 
186  for  the  longest  (see  Figure  V.14).  The  average  size 
for  agency  impact  proposals  was  141  pages  and  for  commercial 

Pages  in  proposals,  excluding  appendicies,  resumes,  supporting 
documents,  and  scope  of  effort  information. 
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FIGURE  V.14  SIZES  OF  THE  PROPOSALS 
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impact  proposals  was  132  pages.  This  is  only  an  approxima- 
tion of  the  volume  of  information  since  the  type  faces, 
spacing,  and  margins  varied. 

Goldplating  of  the  proposals  did  not  seem  to  be  a prob- 
lem. Most  of  the  bidders  did  not  seem  to  be  concerned  with 
providing  the  evaluation  panel  with  information  .in  excess 
of  what  was  specifically  requested  in  the  sub-factors. 

The  primary  exception  to  this  being  one  bidder  who  submitted 
a 23  page  introductory  section.  Bidders  who  wished  to  sup- 
ply evidence  to  support  claims  made  in  their  sub-factor 
responses  did  so  in  the  proposal  appendix  in  the  form  of 
resumes  and  past  project  summaries  or  they  sent  along  re- 
ports from  prior  contracts. 

The  panel  members  were  given  the  proposals  on  a Friday 
and  the  first  factor  team  was  able  to  meet  on  Monday.  Though 
some  panel  members  had  to  read  over  the  weekend  and  at  night 
during  the  next  week,  there  were  no  requests  for  delays 
in  any  of  the  factor  team  meetings.  At  most  there  were 
comments  about  the  pace  of  the  team  meetings  by  some  of 
the  panel  members  who  were  participating  on  a large  number 
of  the  factor  teams.  Since  there  were  fourteen  team  meet- 
ings in  a one  week  period,  people  who  participated  heavily 
could  spend  four  to  six  hours  a day  in  meetings. 

As  one  rough  check  on  the  significance  of  length  of 
response  as  a factor  in  determining  ratings,  the  relation- 
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ships  between  final  score  and  overall  length  of  the  body 
of  the  proposals  are  displayed  in  Figure  V.15.  Inspection 
of  the  graphs  indicates  that  page  length  does  not  seem  to 
be  directly  related  to  score.  Bidders  were  not  rewarded 
for  volume. 

The  issue  of  length  of  response  is  also  considered 
to  be  closely  related  to  the  next  issue  on  whether  the  bid- 
ders understood  what  the  evaluation  panel  looked  at  in  mak- 
ing their  evaluations. 

V.3.2.3  Understanding  basis  for  evaluation.  The 
next  format  issue  is  whether  the  bidders  understood  what 
the  evaluation  panel  looked  at  in  making  their  evaluations. 

Of  concern  here  is  whether  the  bidders  understood  what  infor- 
mation to  include  and  where  in  their  proposals  to  place 
it  so  that  they  would  feel  confident  that  the  panel  would 
use  it  in  evaluating  specific  capabilities.  This  is  related 
to  the  issue  of  size  of  proposals  since  if  the  bidder  is 
unsure  what  the  panel  will  base  their  evaluation  on  that 
there  is  incentive  to  put  everything  possible  into  the  pro- 
posal. To  avoid  receiving  overly  long  proposals  the  govern- 
ment sometimes  uses  page  limitations  when  it  cannot  tell 
the  contractors  exactly  what  it  wants  to  know  (Thompson, 

Note  2).  This  adds  additional  uncertainty  to  the  proposal 
evaluation  process.  If  the  panel  does  not  find  the  informa- 
tion it  is  looking  for  in  a bidder’s  proposal,  it  will  not 
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know  whether  the  information  was  excluded  because  the  bidder 
was  weak  on  the  issue  they  are  evaluating  or  whether  the 
bidder  merely  guessed  wrong  on  the  government’s  priorities 
and  excluded  the  material  in  order  to  meet  the  page  limita- 
t ion . 

The  bidders  were  generally  confident  about  being  able 
to  communicate  their  qualifications  to  the  evaluation  panel. 
A question  with  two  sub-items  was  included  in  the  bidder 
representatives’  questionnaires  to  determine  this.  The 
items  and  responses  are  presented  in  Figures  V.l6  and  V.17. 
Figure  V.l8  lists  responses  to  the  open  ended  questionnaire 
items  which  pertain  to  this  issue. 


Were  you  confident  that  you  had  written  something  to  demon- 
strate yoiir  qualifications  on  all  of  the  capabilities  which 
the  panel  would  be  evaluating? 

Yes-5  Mo-1 

Remarks  by  those  answering  Yes 

o As  we  had  understood  things.  We  did  not  have  confidence 
that  we  know  what  the  evaluators  would  be  thinking  (3R4). 

o We  went  through  point  by  point  (BR1), 

Remark  by  person  answering  No 

0 It  wasn't  clear  what  they  were  trying  to  evaluate.  There 
were  no  a priori  objectives  defined,  therefore,  the  re- 
spondents weren't  responding  to  the  same  project.  It 
wasn't  clear  whether  the  notion  of  cross-referencing  men- 
tioned in  the  RFP  could  be  used.  It  was  questionable 
whether  any  one  reviewer  would  have  a whole  package  to 
look  at  to  cross-reference.  It  wasn't  clear  how  any  panel 


member  would  be  able  to  see  how  any  particular  part  fit 
the  overall  project  (BR5). 

FIGURE  V.16  BIDDER  REPRESENTATIVES ’ RESPONSES  TO  QUESTION 
2a  ON  CONFIDENCE  IN  BEING  ABLE  TO  COMMUNICATE 
QUALIFICATIONS  TO  PANEL 


Were  you  confident  that  the  panel  would  find  the  information 
in  your  proposal  which  was  relevant  to  each  specific  cap- 
ability at  the  time  when  that  capability  was  being  evalu- 
ated? 

Yes  - 4 No  - 2 

Remarks  by  those  answering  Yes 

o But  we  were  trying  to  create  objectivity  where  the  prob- 
lems arising  and  the  sensitivities  that  had  to  be  appealed 
to  were  things  which  were  subjective  (BR4). 

o I think  we  had  a cross-reference  sheet,  therefore,  I would 
feel  confident  that  the  information  would  be  found  (BR1). 

Remarks  by  those  answering  No 

o It  wasn’t  clear  that  cross-referencing  would  work.  It 
wasn't  clear  that  the  reviewers  would  have  the  whole  pack- 
age so  that  they  could  cross-reference  (BR5). 

o With  the  mixed  panel  I didn't  feel  that  they  were  capable. 
I didn't  think  that  they  knew  enough  about  ETIP.  I didn't 
know  all  of  the  people  on  the  panel  but  I remember  at 
the  time  I felt  uneasy.  The  nuances  of  the  program,  I 
had  problems  with  whether  the  average  government  guy  sit- 
ting on  the  panel  would  have  the  discipline  to  go  through 
the  structure  to  make  it  really  carry  (BR2). 

FIGURE  V.17  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
2b  ON  CONFIDENCE  IN  BEING  ABLE  TO  COMMUNICATE 
QUALIFICATION  TO  PANEL 


Response  fr6m  Panel  Question  6 on  Weakest  Aspect  of  the 
Process 


o Trying  to  get  bidders  to  understand  what  you  want  them 
to  put  into  the  procurement  package  so  that  they  could 
be  evaluated  on  their  ability  to  do  the  job,  since  it 
was  new  and  different.  Wanted  to  be  sure  that  we  got 


enough  information  so  we  could  make  a judgment  and  so 
they  know  what  it  was  they  needed  to  provide  in  order 
for  us  to  make  the  judgment  (PM15). 


FIGURE  V.18  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  CONFIDENCE  IN  BEING  ABLE  TO  COMMUNICATE 
QUALIFICATIONS  TO  PANEL 

The  bidder  representatives  generally  felt  confident 

that : 

o they  had  written  something  to  demonstrate  their 
qualifications  on  all  of  the  capabilities  which 
the  panel  would  be  evaluating  (Yes  - 5;  No  - 
1 ) ; and 

o the  panel  would  find  the  information  in  their 
propsal  which  was  relevant  to  each  specific 
capability  at  the  time  when  that  capability 
was  being  evaluated  (Yes  - 4;  No  - 2). 

The  one  bidder  representative  who  answered  No  to  both  items 
either  did  not  read  or  did  not  believe  the  statem.ent  in 
the  RFP  that  explained  that  all  members  of  the  evaluation- 
team  would  have  access  to  the  complete  proposal.  The  other 
bidder  representative  who  answered  No  to  the  second  item 
did  not  feel  that  the  people  on  the  panel  would  be  disci- 
plined enough  to  follow  the  process  which  was  described 
in  the  RFP. 

Information  provided  in  item  1c  of  the  panel  members’ 
questionnaire  relates  to  this  issue.  The  item  asked  panel 
members  whether  they  were  readily  able  to  locate  information 
in  the  proposals  which  was  relevant  to  the  issues  being 
evaluated.  The  response  was  Yes  - 14,  No  - 1 (see  Figure 
V.12).  The  five  respondents  who  provided  additional  rem.arks 


indicated  that  the  only  times  they  had  problems  were  when 
the  bidders  did  not  cross-reference  properly  or  otherwise 
follow  the  directions. 

V.3.2.4  Differences  in  uncertainty  among  bidders. 

The  final  format  issue  concerns  the  panel’s  ability  to  deal 
effectively  with  differences  in  uncertainties  among  bidders 

t 

during  the  evaluation.  There  were  two  ways  in  which  this 
became  an  issue.  One  was  that  bidders  who  had  worked  with 
ETIP  before  had  a greater  prior  knowledge  of  the  program, 
its  projects,  and  the  interests  of  its  staff.  They  would 
therefore  be  able  to  write  and  speak  more  knowledgeably 
about  program  content  than  bidders  who  had  no  prior  experi- 
ence. The  other  way  was  a corollary  of  this.  The  panel 
would  have  more  information  about  bidders  who  had  worked 
for  ETIP  before  than  those  who  had  not.  Its  ability  to 
rate  capabilities  on  some  of  the  sub-factors  for  bidders 
who  had  been  past  contractors  would  be  greater  than  for 
those  who  had  not.  This  relates  closely  to  the  issue  in 
section  V.3.7  on  whether  there  were  problems  in  dealing 
with  the  lack  of  restrictions  on  admissible  information. 

The  PFE  process  seemed  to  be  successful  in  dealing 
with  both  of  these  types  of  uncertainties.  It  provided 
for  an  evaluation  based  on  present  or  potential  capabilities 
rather  than  on  present  knowledge.  It  also  provided  mechan- 


isms  for  using  all  useful  information  about  bidders  where 
it  was  available  and  for  obtaining  clarifications  where 
it  was  not.  One  outcome  measure  here  is  that  one  of  the 
two  contracts  was  won  by  a bidder  who  had  not  previously 
worked  for  ETIP. 

There  are  two  structural  mechanisms  of  PFS  which  dealt 
with  the  problem  of  enabling  bidders  without  prior  program 
experience  to  compete  effectively.  One  is  that  the  process 
provided  as  much  information  about  ETIP’s  plans  as  was  pos- 
sible. The  other  is  that  the  evaluation  was  based  upon 
the  bidder’s  understanding,  approach,  and  present  or  poten- 
tial capabilities  to  deal  with  specific  management  and  tech 
nical  problems  rather  than  upon  a proposed  design  requiring 
integrating  detailed  knowledge  about  the  program  into  the 
proposal . 

Information  about  ETIP  and  the  proposed  evaluation 
systems  was  provided  in  the  form  of  the  two  bidders'  brief- 
ings and  the  supporting  written  materials  and  bibliography 
available  with  the  solicitation  document.  The  goal  was 
to  provide  information  about  both  areas  of  high  and  low 
certainty  concerning  the  proposed  systems  and  to  allow  pro- 
spective bidders  to  obtain  clarifications  through  questions 
The  one  problem  which  arose  in  providing  information  was 
ETIP's  inability  to  distribute  the  COTR ' s project  plans. 


Since  the  plans  had  not  been  approved  by  NBS  administration 
until  after  the  second  briefing,  they  were  unavailable  for 
distribution  to  bidders.  If  they  had  been  available,  they 
would  have  provided  some  additional  written  detail  on  the 
projects  to  support  the  information  presented  during  the 
first  bidder’s  briefing. 

By  evaluating  proposals  based  on  responses  to  problems 
which  the  government  felt  were  going  to  be  critical,  rather 
than  on  proposed  systems  designs,  ETIP  minimized  the  degree 
to  which  prior  knowledge  of  the  program  would  be  critical. 
Also,  the  criteria  for  assessing  capability  was  whether 
the  bidder  had  a present  or  potential  capability.  If  a 
bidder  did  not  have  a specific  in-house  capability  but  recog- 
nized the  need  for  such  a capability  and  had  an  approach 
for  obtaining  the  capability,  he  was  not  penalized.  Chapter 
IV  presented  an  example  of  a situation  where  a rating  was 
resolved  in  favor  of  a bidder  based  on  a potential  capabil- 
ity (section  IV. 10. 8). 

There  were  two  primary  mechanisms  for  dealing  with 
the  panel’s  differences  in  uncertainty  about  the  various 
bidders.  The  first  was  the  question  form  for  obtaining 
specific  information  to  resolve  uncertainties.  The  second 
was  the  process  of  making  notes  in  the  record  on  specific 
uncertainties  and  obtaining  subsequent  oral  and  written 
clarifications  following  the  panel  meetings. 


In  measurement  terms  the  amount  of  uncertainty  can 
be  thought  of  as  the  confidence  interval  around  the  panel’s 
estim.ate  of  the  bidder’s  score.  On  bidders  for  which  there 
is  less  uncertainty  the  confidence  limits  are  tighter. 

The  clarification  process  can  be  thought  of  as  a process 
for  tightening  the  confidence  intervals  in  the  areas  where 
there  was  the  greatest  amount  of  uncertainty. 

The  question  form  proved  to  be  of  minimum  usefulness. 
Only  two  forms  were  used  and  they  were  submitted  to  obtain 
sections  which  were  missing  from  two  of  the  proposals. 

The  process  of  submitting  question  forms  to  obtain  clarifica- 
tions was  used  in  place  of  having  each  bidder  make  an  oral 
presentation  on  their  proposals  to  the  panel.  This  was 
done  since  it  was  uncertain  how  many  bids  would  be  received 
and  it  was  thought  that  there  might  be  scheduling  difficult- 
ies in  having  the  panel  hear  the  presentations.  Since  there 
were  no  oral  presentations  to  the  panel  and  since  the  ques- 
tion form  was  not  used  for  clarification,  the  burden  for 
clearing  up  uncertainties  was  placed  on  oral  and  written 
clarifications  after  the  panel  meetings. 

The  oral  and  written  clarification  process  worked  quite 
well.  If  the  team  had  difficulty  reaching  agreement  on 
a single  score  because  of  uncertainty  about  the  bidder, 
a note  was  included  in  the  written  record  indicating  that 
the  specific  area  of  uncertainty  needed  to  be  investigated 


during  the  oral  clarifications.  This  usually  happened  when 
there  was  strong  disagreement  between  team  members  concern- 
ing how  to  interpret  something  which  a bidder  stated  in 
his  proposal.  Someone  on  the  team  would  usually  propose 
a score  which  the  majority  of  the  panel  seemed  to  be  favor- 
ing and  suggest  that  the  strong  disagreement  be  resolved 
by  adding  a note  which  would  require  clarification  prior 
to  negotiations.  An  example  of  this  was  given  in  chapter 
IV  in  the  description  of  the  first  factor  team  meeting. 

The  example  tells  of  a disagreement  over  whether  a bidder 
did,  in  fact,  have  the  in-house  capability  in  instrument 
design.  The  issue  was  resolved  by  giving  the  bidder  an 
acceptable  score  (seven  or  above)  and  adding  a note  to  the 
record  to  verify  this  score  (section  IV. 10.2).  All  of  the 
notes  from  the  record  were  compiled  and  used  to  prepare 
questions  for  the  oral  clarifications. 

One  panel  member,  in  responding  to  item  10  on  the  panel 
questionnaire  (see  Figure  V.19),  indicated  that  he  would 
like  to  have  had  information  from  reference  checks  available 


Responses  From  Panel  Question  10  on  Any  Other  Aspects  of 

the  Process  which  You  Wish  to  Get  Out 

o Checking  of  bidders*  references  should  be  organized 

more  thoroughly  in  advance  of  the  panel  meetings  (PM12). 

o The  orals  were  very  important.  Since  I didn’t  partic- 
ipate I don’t  know  how  well  they  worked.  They  shouldn’t 
be  considered  pro  forma.  It’s  your  last  chance  to 
ask  tough  questions  and  see  the  answers  (PM10). 

FIGURE  V.19  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  DIFFERENCES  IN  UNCERTAINTY  AMONG  BIDDERS 
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during  the  factor  team  meetings.  The  way  the  process  worked 
was  that  references  (people  listed  under  Factor  I as  having 
knowledge  of  the  bidder's  past  performance)  were  checked 
after  the  team  meetings  and  before  the  oral  clarifications. 
The  author  believes  that  this  information  and  information 
which  would  have  been  available  from  orals  prior  to  the 
panel  meeting  would  have  enabled  the  panel  to  tighten  the 
confidence  intervals  around  its  scores  and  take  some  of 
the  burden  off  of  the  subsequent  oral  clarification  sessions. 

V. 3.2.5  Communication  of  nature  of  project  and  required 
skills.  The  first  content  issue  under  the  Effective  Com- 
munications category  is  whether  PFE  communicated  to  bidders 
what  it  was  the  government  wanted  to  buy?  Of  importance 
here  is  whether  the  bidders  understood  the  general  nature 
of  the  projects  well  enough  so  that  they  could  effectively 
respond  to  the  sub-factors.  A level  of  understanding  which 
conveyed  both  the  skills  required  and  the  context  for  re- 
sponding to  the  sub-factors  in  terms  of  the  project  objec- 
tive was  important.  Related  to  this  is  that  it  was  import- 
ant that  the  requirement  for  the  projects  which  were  conveyed 
in  the  RFP  were  the  requirements  which  the  government  ac- 
tually undertook  after  the  contracts  were  signed,  i.e., 
the  actual  projects  be  the  projects  on  which  the  source 
selection  was  based.  Often  the  criteria  for  competitive 
selection  turn  out  to  have  little  relationship  to  the  pro- 


ject  which  the  successful  bidder  ends  up  undertaking  (see 
Sect  ion  II . 3 • 5 ) . 

The  extent  to  which  the  process  was  able  to  communicate 
t;he  general  nature  of  the  projects  and  the  required  skills 
is  somewhat  uncertain.  A question  with  two  sub-items  was 
included  in  the  bidder  questionnaire  to  obtain  information 
on  this.  The  items  and  responses  are  presented  in  Figures 
V.20  and  V.21.  Figure  V.22  presents  a response  to  the  open 
ended  questionnaire  items  which  pertains  to  this  issue. 

The  bidders  were: 

o split  on  whether  they  felt  that  the  information 
provided  by  ETIP  allowed  them  to  assess  the 
skills  which  would  be  required  to  do  the  work 
( Yes  - 3 ; No  - 3 ) ; and 

o almost  unanimous  in  feeling  that  ample  informa- 
tion was  provided  in  the  briefings  and  support- 
ing documents  so  that  they  were  confident  that 
they  had  sufficiently  related  the  sub-factor 
issues  to  the  objectives  of  the  projects  (Yes  - 
5;  No  - 1). 

Remarks  under  the  first  item  (question  3a,  Figure  V.20) 
indicate  that  some  of  the  bidders  did  not  feel  that  the 
objectives  of  the  project  and  ETIP  were  clearly  defined. 
Another  bidder  felt  that  he  did  not  know  enough  about  the 
shortcomings  of  the  people  managing  ETIP. 

The  latter  eoncern  is  one  for  which  it  would  be  dif-  • 
ficult  to  fault  the  structure  of  content  of  the  source  selec- 
tion process.  It  is  doubtful  that  any  process  could  effec- 
tively provide  information  on  the  management  styles  of  pro- 


Did  the  information  provided  by  ETIP  allow  you  to  assess 

the  skills  which  would  be  required  to  do  the  work? 

Yes  - 3 Mo  - 3 

Remarks  by  those  answering;  Yes 

o Generally.  I didn’t  think  ETIP  exactly  knew  v;hat  it  wante 
The  REP  was  as  good  as  it  could  be  under  the  circumstances 
It  was  the  nature  of  the  program.  ETIP  was  very  into 
process  and  could  have  discussed  more  of  the  substance 
of  the  program.  Especially  the  politics  and  communica- 
tions skills  to  handle  a job  like  that  (BR2). 

o We  felt  the  information  was  there  to  assess  the  skill 
requirements  but  we  had  a feeling  that  more  skills  were 
asked  for  than  were  really  required.  That  may  have  been 
due  to  a misinterpretation  of  the  RFP  which  could  be  con- 
strued as  poor  wording  in  the  RFP  or  maybe  not  (3R1). 

Remarks  by  those  answering  No 

o There  weren't  any  well  defined  project  objectives.  Since 
there  was  no  general  task,  one  couldn't  show  what  selec- 
tion of  skills  was  necessary  (BR5). 

o We  needed  a clearer  picture  of  what  ETIP  hoped  to  accom- 
plish. It  didn't  have  anything  to  do  with  the  factor/sub- 
factor issues.  It  was  that  the  mission  of  ETIP  was  not 
clear  ( BR4 ) . 

0 We  needed  a better  way  to  anticipate  the  ineptitude  on 
the  part  of  ETIP  management  (BR6). 


FIGURE  V.20  BIDDER  REPRESENTATIVES ' RESPONSES  TO  QUESTION 
3a  ON  WHETHER  THE  RFP  COMMUNICATED  WHAT  THE 
GOVERNMENT  WANTED  TO  BUY 


Was  ample  information  provided  in  the  briefings  and  support- 
ing documents  so  that  you  were  confident  that  you  had  suf- 
ficiently related  the  sub-factor  issues  to  the  objectives 
of  the  projects? 

Yes  - 5 No  - 1 

Remarks  by  those  answering  Yes 

o Qualified  by  that  I didn’t  feel  the  true  feelings  were 
reflected.  You  were  trying  to  take  an  objective  approach 
where  in  the  end  subjective  feelings  had  a lot  to  do  with 
it  (BR4). 

o Thompson  was  too  much  into  process.  I felt  he  didn’t 
really  know  government  and  it  showed.  More  substance 
and  less  process  (BR2). 

Remark  by  person  answering  No 

o There  were  no  good  objectives  or  purpose  other  than  that 
ETIP  was  politically  in  trouble  at  the  time  (BR5). 

FIGURE  V.21  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
3b  ON  WHETHER  THE  RFP  COMMUNICATED  WHAT  THE 
GOVERNMENT  WANTED  TO  BUY 


Response  to  Bidder  Question  13  on  Any  Other  Aspects  of  the 
Process  Which  You  Wish  to  Get  Out 


o There  seemed  to  be  a great  deal  of  jargon  in  the  RFP. 

We  spent  quite  a bit  of  time  trying  to  figure  out  what 
you  were  really  saying  (BR1). 

FIGURE  V.22  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 

TO  WHETHER  THE  RFP  COMMUNICATED  WHAT  THE  GOVERN- 
MENT WANTED  TO  BUY 


gram  personnel.  This  seems  to  be  the  type  of  information 
which  is  somewhat  available  to  bidders  through  contact  with 
others  in  the  professional  community. 

The  former  concern  of  lack  of  clarity  of  objectives 
is  difficult  to  reconcile  with  the  near  unanimous  response 
to  the  second  item  (question  3b,  Figure  V.21)  which  asked 
whether  the  information  presented  was  sufficient  to  relate 
the  sub-factor  issues  to  the  project  objectives.  In  part 
one  bidder  (a  successful  bidder)  may  be  responding  to  prob- 
lems which  occurred  after  the  source  selection  was  complete 
and  work  began  on  the  projects. 

The  problem  of  the  project  actually  undertaken  bearing 
little  relationship  to  the  project  described  in  the  RF? 
was  eliminated  by  using  PFE.  Since  the  source  selection 
material  specified  both  the  areas  of  certainty  (e.g.,  eval- 
uation systems  for  commercial  and  agency  impacts  of  procure 
ment  experiments  were  required)  and  uncertainty  (e.g.,  the 
number  of  evaluation  systems,  their  eventual  owners,  and 
the  specific  projects  to  be  evaluated  during  the  develop- 
ment) the  bidders  were  not  responding  with  specific  solu- 
tions to  objectives  which  would  change. 

V.3.2.6  Appropriateness  of  the  Specific  Sub-factors. 
The  next  content  issue  deals  with  the  appropriateness  of 
the  sub-factors  which  were  selected.  One  of  the  PAA  staff 
members  who  reviewed  a draft  of  the  RFP  indicated  that  he 
felt  that  many  of  the  sub-factors  under  Management  and  Re- 


ports  & Reviews  would  not  be  useful  in  the  evaluation  (see 
Section  IV. 6).  Another  related  concern  is  whether  there 
were  any  significant  issues  which  were  not  included  as  sub- 
factors. The  first  concern,  if  true,  would  result  in  some 
waste  of  resources  by  bidders  in  responding  to  the  RFP  and 
by  the  government  in  evaluating  the  RFP.  The  second  concern 
would  be  more  serious,  since  it  could  result  in  selecting 
an  unqualified  bidder  by  not  evaluating  some  critical  issue. 

Both  panel  members  and  bidder  representatives  largely 
felt  that  the  selection  of  sub-factors  was  appropriate. 

A question  with  two  sub-items  was  included  in  both  the  panel 
and  bidder  questionnaires  to  obtain  reaction  to  the  specific 
set  of  sub-factors  which  was  used.  The  items  and  responses 
to  them  are  presented  in  Figures  V.23  to  V.26.  Figure  V.27 
presents  a response  by  a bidder  to  the  open  ended  question- 
naire items  which  pertains  to  this  issue. 


Did  you  feel  the  sub-factors  encompassed  most  of  the  issues 
which  would  be  critical  to  designing  the  evaluation  systems 
which  ETIP  desired? 

Yes  - 12  No  - 2 Don't  know  - 1 

Remark  by  a person  answering  Yes 

o If  you  cover  everything  you  don't  miss  much.  The  sub- 
factors covered  just  about  everything,  probably  too  much 
(PM4). 

Remarks  by  those  answering  No 

o Economic  issues  were  given  insufficient  weight  in  the 
commercial  impact  solicitation  (PM2). 
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0 A bidder's  recognition  of  the  constraints  of  reality  of 
evaluation  was  not  adequately  covered  (PM3). 

Remark  by  person  answering  Don't  know 

0 I assumed  ETIP  knew  best  what  it  wanted  (PM6). 

FIGURE  V.23  PANEL  MEMBERS'  RESPONSES  TO  QUESTION  2a  ON 
APPROPRIATENESS  OF  THE  SUB-FACTORS 


Did  you  feel  the  sub-factors  encompassed  most  of  the  issues 

which  would  be  critical  to  designing  the  evaluation  systems 

which  ETIP  desired? 

Yes  - 4 No  - 2 

Remark  by  a person  answering  Yes 

o Most  of  the  sub-factors  were  O.K.  but  they  weren't  really 
important.  More  emphasis  should  have  been  placed  on  the 
problems  of  working  with  other  agencies.  You  had  a gen- 
erally inexperienced  staff  (BR2). 

Remarks  by  those  answering  No 

o There  weren't  any  objectives,  therefore  it's  hard  to  say 
what  issues  derive  from  them  (BR5). 

o Failure  on  the  part  of  ETIP  management  to  com.prehend  the 
process  by  which  innovations  take  place  (BR6). 

FIGURE  V.24  BIDDER  REPRESENTATIVES ' RESPONSES  TO  QUESTION 
4a  ON  APPROPRIATENESS  OF  THE  SUB-FACTORS 


Did  you  feel  that  many  of  the  sub-factors  dealt  with  issues 
unimportant  for  designing  the  evaluation  systems  which  ETIP 
desired? 

Yes  - 1 No  - 13  Don't  know  - 1 

Remark  by  person  answering  Yes 

I 

0 Many  of  the  agency  and  system-related  factors  are  either 
of  marginal  importance  or  can  only  be  developed  through 
an  inductive  process  over  time,  not  through  a deductive 
process  which  was  the  strategy  behind  the  evaluation  system 
(PM2). 


Remarks  by  those  answering  No 


o Comments  from  people  who  saw  it  or  bid  on  it  felt  it  was 
complex  to  respond  to.  But  it  was  simple  to  respond  to 
if  you  read  it  and  understood  the  format.  It  was  not 
complicated.  It  was  a question  of  devoting  time  to  it 
(PM10) . 

o A few  seemed  superfluous  (PM1). 

o A lot  of  things  are  important  but  not  important  enough 
to  be  set  out  as  separate  sub-factors.  You  could  subsume 
the  sub-factors  under  the  factors.  You’re  cutting  the 
pie  too  fine  (PM4). 

o But  "design,"  in  the  formal  sense,  received  too  much  empha- 
sis (PM3 ) . 

Remark  by  person  answering  Don’t  know 

0 Didn’t  feel  I had  enough  information  about  the  total  pro- 
ject to  judge  the  importance  of  the  issues  covered  (PM6). 

FIGURE  V.25  PANEL  MEMBERS’  RESPONSES  TO  QUESITON  2b  ON 
APPROPRIATENESS  OF  THE  SUB-FACTORS 


Did  you  feel  that  many  of  the  sub-factors  dealt  with  issues 
unimportant  for  designing  the  evaluation  systems  which  ETIP 
desired? 

Yes  - 2 No  - 4 

Remarks  by  those  answering  Yes 

o I felt  yes  at  the  time  but  I really  don’t  remember  why 
(BR5). 

o I don’t  remember  which  ones  (BR1). 

Remark  by  a person  answering  No 

o There  was  a lack  of  balance  and  some  of  them  could  have 
been  handled  under  more  general  topics.  A more  substan- 
tive discussion  could  have  given  better  balance  (BR2). 

FIGURE  V.26  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
4b  ON  APPROPRIATENESS  OF  THE  SUB-FACTORS 


Response  to  Bidder  Question  10  on  the  Strongest  Aspect  of 
the  Process 


2 


o Its  balance  and  its  all  inclusiveness.  It  was  an  awesome 
job  to  take  it  initially.  After  analysis,  it  was  still 
awesome  but  you  could  see  where  it  was  going.  It  was 
clear  and  the  requirements  were  in  balance.  Nothing  was 
missing  ( BR3 ) . 

FIGURE  V.27  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  APPROPRIATENESS  OF  THE  SUB-FACTORS 


The  panel  members  generally  felt  that  the  subfactors: 

o encompassed  most  of  the  critical  issues  for 
designing  the  evaluation  system.s  (Yes  -12; 

No  - 2;  Don’t  know  - 1);  and 

o did  not  deal  with  issues  unimportant  for  design- 
ing the  evaluation  systems  (Yes  they  did  deal 
with  unimportant  issues  - 1 ; No  they  did  not  - 
1 3 ; Don ’ t know  - 1 ) . 

The  bidders  were  more  evenly  divided.  However,  they  also 
felt  that  the  sub-factors: 

o encompassed  the  critical  issues  (Yes  - 4 ; No  - 
2 ) ; and 

o did  not  deal  with  unimportant  issues  (Yes  they 
did  - 2;  No  they  did  not  - 4). 

No  trends  appear  in  the  remarks  of  the  respondents.  How- 
ever, it  is  interesting  to  note  the  not  unexpected  contrast 
in  perspective  between  two  people  responding  to  the  same 
RFP.  Bidder  representative  2 (under  question  4b,  Figure 
V.26)  indicates  that  there  was  a lack  of  balance  among  the 
sub-factors,  while  bidder  representative  3 (under  question 
10,  Figure  V.27)  indicates  that  the  balance  and  all  inclu- 
siveness was  the  strongest  aspect  of  the  process. 


V.3.2.7  Parroting . A final  content  of  communication 
issue  is  concerned  with  whether  the  bidders  supply  infor- 
mation in  their  proposals  which  goes  beyond  problem  state- 
ments or  potential  solutions  which  are  contained  in  docu- 
ments supplied  as  part  of  the  solicitation.  Parroting  of 
information  from  the  solicitation  is  a problem  when  it  is 

t 

done  to  the  exclusion  of  information  which  would  enable 
the  panel  to  evaluate  the  bidders.  Some  ETIP  staff  had 
raised  this  issue  prior  to  the  release'  of  the  RFP  (Thompson, 
Note  3 ) • 

There  did  not  seem  to  be  any  significant  problems  with 
parroting.  Since  the  bidders  were  not  given  a specific 
system  design  to  respond  to,  there  was  no  proposed  solution 
to  echo  back.  The  bidders’  responses  addressed  the  issues 
which  were  raised  in  the  sub-factor  statements  or  the  illus- 
trative questions.  In  several  cases  the  team  found  that 
the  information  which  was  provided  under  a specific  sub- 
factor section  of  the  proposal  was  sparse  and  the  bidder 
had  not  cross  referenced  other  sections  of  his  proposal. 

If  a panel  member  had  read  information  in  other  parts  of 
the  proposal  which  seemed  relevant  to  the  sub-factor  being 
evaluated,  this  information  was  presented  to  the  panel  for 
consideration  (see  Section  IV. 10. 6). 

V.3.3  Sensitivity  and  Selectivity  in  Obtaining  Bids 

The  category  of  sensitivity  and  selectivity  covers 
issues  relating  to  whether  ETIP  was  able  to  obtain  an  accept- 


able  number  of  responsive  bids. 


The  two  issues  under  this 


category  are: 

1 . Were  enough  responsive  bids  received  to  allow 
for  a confident  evaluation? 

2.  Did  the  process  enable  prospective  bidders 
to  self-select  effectively  on  bidding? 

Were  a large  number  of  non-respons i ve  bids 
received? 

Did  a lot  of  qualified  firms  not  bid? 

V.3.3.1  Sensitivity.  The  issue  of  sensitivity  was 
one  of  how  to  attract  bidders  who  might  have  the  required 
capabilities.  The  program's  budget  was  small  compared  to 
other  federal  programs  offering  evaluation  or  systems  con- 
tracts. It  was  therefore  not  attractive  as  a continuing 
source  of  funds.  Also,  it  did  not  have  the  visibility  in 
the  professional  evaluation  community  that  a bidder  might 
desire.  The  program  had  begun  to  establish  a poor  track 
record  in  contracting  by  cancelling  its  last  RF?  for  an 
evaluation  contract  after  the  proposals  were  evaluated. 

One  of  the  concerns  of  people  at  ETIP  both  prior  to  and 
after  the  writing  of  the  RFP  was  whether  enough  responsive 
bids  would  be  received  to  allow  for  a confident  evaluation 
and  selection  of  qualified  bidders  (see  Section  IV. 3). 

Also,  when  the  RFP  was  circulated  for  review,  a PAA  staff 
person  felt  that  it  was  so  confusing  that  it  would  scars 
off  good  bidders  (see  Section  IV. 6). 

The  responses  of  the  panel  members  suggest  that  enough 
bids  were  received  to  allow  confidence  in  the  accuracv  of 


the  evaluation.  Whether  or  not  any  firms  with  greater  qua- 
lifications did  not  bid  is  uncertain. 

An  item  in  the  panel  questionnaire  asked  respondents 
whether  being  able  to  rate  more  proposals  would  have  in- 
creased their  confidence  in  the  accuracy  of  the  scores  which 
were  assigned.  For  the  agency  impact  contract  the  answers 
were;  Yes  - 1 ; No  - 10.  For  the  commercial  impact  contract 
they  were:  Yes  - 3 j No  - 7 (see  Figure  V.2S). 


Did  you  feel  that  being  able  to  rate  more  proposals  would 
have  increased  your  confidence  in  the  accuracy  of  the  scores 
which  were  assigned? 

a)  Agency  Impact 

Yes  - 1 No  - 10 

Remark  by  person  answering  Yes 

o About  a dozen  in  all.  If  you  were  evaluating,  you  need 
to  see  a number  of  selections.  It  was  more  of  an  issue 
for  those  participating  in  only  one  factor  team  (PM1). 

b)  Commercial  Impact 

Yes  - 3 No  - 7 

Remarks  by  those  answering  Yes 

o About  two  more.  You  want  to  weigh  them  to  be  sure 
you’ve  seen  the  best  that’s  around  (PM12). 

o About  two  more  (PM7). 

0 At  least  three  more  and  maybe  ten  total  (PM15). 

Remark  by  a person  answering  No 

o I could  have  picked  the  winners  without  any  scoring 
(PM4). 

FIGURE  V.28  PANEL  MEMBERS’  RESPONSES  TO  QUESTION  3 WHICH 
RELATES  TO  SENSITIVITY 


The  responses  for  the  commercial  impact  contract,  for 
which  three  proposals  were  received,  seem  to  indicate  that 
it  is  possible  to  scale  bidders  with  the  process  when  only 
a small  number  of  proposals  are  re.ceived.  However,  the 
confidence  that  some  of  the  panel  members  had  in  their  eval- 
uation on  the  commercial  impact  contract  may  have  been  in- 
creased by  having  been  able  to  scale  two  of  the  bidders 
during  the  agency  impact  evaluation  (i.e.,  two  out  of  the 
three  had  bid  on  agency  impact  as  well  and  had  already  been 
evaluated  once  on  most  of  the  factors). 

Even  though  enough  responsive  bids  were  received  to 
allow  for  confidence  in  the  evaluation,  there  remains  the 
question  of  whether  any  better  qualified  firms  did  not  bid 
and  if  so  whether  this  was  a function  of  the  ?FE  proposal. 
This  question  had  been  raised  by  several  people  following 
the  source  selection. 

The  problem  with  this  question  is  that  there  is  prob- 
ably no  way  to  answer  it.  Since  there  was  not  a well  estab- 
lished group  of  firms  with  a track  record  in  implementing 
evaluation  systems  (one  of  the  conditions  for  using  ?FE), 
it  was  not  possible  to  assess  a priori,  or  without  evalua- 
tion, who  the  more  qualified  contractors  were.  People  ob- 
served that  some  of  the  firms  that  they  expected  to  bid 
did  not.  This,  however,  may  be  a function  of  many  variables 
including  the  novelty  of  the  source  selection  method  and 


effective  self-selec t ion 


V.3.3.2  Selectivity . The  goal  of  self-selection  is 


to  allow  those  prospective  bidders  who  would  not  be  quali- 
fied to  do  the  work  to  determine  this  themselves  and  not 
invest  in  bidding.  In  this  way  resources  of  both  the  bid- 
ders and  the  government  are  saved.  The  literature  has  dis- 
cussed the  waste  which  occurs  when  many  unqualified  bidders 
submit  proposals  (see  Section  II. 3. 5).  This  issue  relates 
to  the  issues  on  resource  requirements  which  are  described 
in  section  V . 3 . 1 2 . 

The  RFP  seemed  to  be  relatively  effective  on  the  issue 
of  self-selection.  Though  bidders  were  evaluated  by  the 
teams  as  being  unacceptable  on  specific  sub-factors  and 
some  bidders  were  unacceptable  on  a large  number  of  sub- 
factors, the  solicitation  did  not  result  in  a large  number 
of  obviously  unacceptable  proposals.  Those  which  ended 
up  as  unacceptable  required  consideration  by  the  teams  on 
each  sub-factor  to  determine  this  (see  related  responses 
to  open-ended  questions  displayed  in  Figure  V.29). 

Response  to  Panel  Question  5 on  the  Strongest  Aspect  of 
the  Process 

o It  weeded  out  the  trivial  bidders.  You  don't  want  the 
casual  people  taking  up  your  time.  Those  who  responded 
clearly  were  sincere  (PM10). 

Response  to  Panel  Question  10  on  Any  Other  Aspects  of  the 
Process  Which  You  Wish  to  Get  Out 

o It  provides  information  to  potential  bidders  for  self- 
selection (PM13). 

FIGURE  V.29  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  SELECTIVITY 
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Self-selection  may  have  occurred  by  each  of  two  mechan- 
isms. One  was  by  prospective  bidders  balancing  the  objec- 
tives of  the  project  against  the  objectives  of  their  firm, 
and  the  resources  available  to  their  firm.  The  other  was 
by  determining  whether  they  would  be  able  to  respond  effec- 
tively to  the  RFP . The  author  has  heard,  third  hand,  about 
both  types  of  selections  by  firms  which  some  ETIP  staff 
had  expected  would  bid.  The  first  being  firms  where  manage- 
ment decided  that  the  project  did  not  match  their  current 
or  long  term  objectives.  The  second  being  fir.ms  which  were 
upset  about  the  form  of  the  RFP  and  did  not  wish  to  invest 
in  responding.  One  possibility  is  that  some  of  the  firms 
which  were  upset  about  the  form  of  the  RFP  would  have  been 
equally  upset  about  the  complexity  of  the  requirements  of 
the  projects  if  they  were  under  contract. ° 

V.3.4  Assessment  of  Bidders’  Capability 
The  category  of  Assessment  of  Bidders’  Capability  is 
concerned  with  issues  of  whether  PFE  enabled  an  effective 
evaluation  of  both  management  and  technical  capabilities 


°0n  a later  procurement  using  PFE,  one  firm  which  de- 
clined to  bid  was  quite  open  about  their  opposition  to  the 
form  of  the  RFP.  They  listed  numerous  problems  which  they 
found  with  it.  A staff  member  of  this  firm  later  indicated 
that  in  his  opinion  the  firm  was  not  .set  up  to  handle  the 
type  of  unstructured  system  development  work  which  was  re- 
quired and  that  the  management  was  upset  because  they  wanted 
a contract  to  work  in  the  substantive  area  of  the  project 
in  the  manner  which  they  felt  was  required. 


and  whether  this  evaluation  could  be  related  to  subsequent 


performance.  This  was  an  important  problem  since  there 
was  no  known  track  record  of  developing  evaluation  systems 
in  the  contracting  community.  The  meaning  here  of  effective 
evaluation  is  that  the  process  be  successful  in  enabling 
an  ordering  of  bidders  from  best  to  worst,  with  demarcations 
for  acceptability  and  unacceptability,  and  that  this  ordering 
be  based  on  the  actual  capability  of  the  bidders  proposed 
team  rather  than  on  the  capability  of  the  proposal  writers. 
The  question  of  subsequent  performance  is  concerned  with 
whether  an  evaluation  in  terms  of  capability  for  dealing 
with  problems  known  to  exist  is  effective  in  predicting 
a bidder’s  actual  performance. 

Three  issues  are  included  in  this  section.  They 


are ; 


1 . Did  PFE  provide  for  an  effective  evaluation 
of  management  capability? 

2.  Did  PFE  provide  for  an  effective  evaluation 
of  technical  capability? 

3.  Did  PFE  anticipate  the  problems  which  were 
known  to  exist  in  doing  evaluations  and  in 
developing  systems? 

Did  the  criteria  provide  for  an  assessment 
of  the  relationship  between  what  was  proposed 
and  later  performance? 

Was  the  award  based  upon  proposals  which 
would  have  some  relationship  to  the  approach 
subsequently  used? 

Was  the  material  provided  by  the  successful 
contractors  useful  or  not  to  the  PAAs? 
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V.3.4.1  Evaluation  of  management  capability.  The 
need  for  source  selection  procedures  to  be  able  to  deal 
more  effectively  with  management  capabilities  is  evident 
in  both  the  evaluation  and  the  systems  literature.  The 
evaluation  literature  indicates  an  insensitivity  of  evalu- 
ators to  problems  of  liaison  and  scheduling  (e.g.,  Patton, 
et  al.,  1977)  and  some  of  the  systems  literature  indicates 
the  need  to  develop  better  capability  for  evaluating  manage- 
ment during  source  selection  (e.g.,  Helman  4 Taylor,  1975). 
This  issue  primarily  examines  whether  ETIP  was  able  to  dif- 
ferentiate among  bidders  on  the  various  management  issues 
which  were  included  as  sub-factors.  The  question  of  the 
relationship  between  these  ratings  and  future  performance 
is  discussed  under  section  V.3.4.3. 

The  author  believes  that  PFE  provided  a strong  basis 
for  evaluating  management  capability.  The  discussion  by 
the  factor  teams  of  the  management  sub-factors  was  exten- 
sive. Much  of  the  time  was  taken  to  draw  out  the  differ- 
ences between  the  management  requirements  on  these  projects 
and  general  project  management  requirements.  The  teams 
used  the  written  sections  in  the  proposal  and  all  supporting 
evidence  (resumes,  project  summaries  and  information  from 
panel  members  who  were  familiar  with  the  bidders)  to  assess 
understanding,  approach  and  capability.  For  the  most  part 
the  teams  were  able  to  come  to  agreement  on  unique  scores 
for  each  bidder.  However,  there  was  an  important  management 
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sub-factor  which  proved  difficult  for  one  of  the  teams  to 
measure.  On  sub-factor  1.2,  Importance  of  Program  to  the 
Bidder,  the  agency  impact  contract  factor  team  assigned 
a 6 to  all  of  the  bidders.  The  team  felt  that  from  the 
information  available  to  them  there  was  no  basis  for  distin- 
guishing among  the  bidders.  The  score  of  six  was  assigned 
because  the  team  felt  that  this  issue  needed  to  be  explored 
during  clarifications  with  any  bidders  considered  competi- 
tive . 

Sub-factor  1.2  was  considered  by  some  to  be  one  of 
the  most  important  in  predicting  the  success  of  the  projects. 
Therefore,  a considerable  amount  of  attention  was  devoted 
to  it  during  clarifications.  The  bidders  were  asked  direct- 
ly about  what  assurances  they  could  give  of  top  management 
support  of  the  project.  Their  responses  to  other  questions 
were,  in  part,  assessed  to  determine  what  they  felt  the 
primary  objectives  of  the  project  were  and  what  they  saw 
as  their  role  in  the  systems  developments.  It  was  important 
for  ETIP  to  know  if  a bidder  had  strong  ideas  about  what 
the  primary  objectives  were  and  if  so  whether  the  bidder 
would  subordinate  ETIP’s  objectives  to  its  own.  The  oral 
clarification  session  proved  very  useful  in  enabling  ETIP 
to  more  confidently  assess  these  issues  and  assign  scores. 
However,  there  was  recognition  that  the  confidence  intervals 


on  some  of  the  bidders  was  still  relatively  large,  especi- 
ally in  trying  to  predict  commitment  into  the  future. 

The  clarifications  on  management  also  allowed  ETIP 

to  determine  which  people  on  the  bidders’  teams  held  the 

understandings  which  the  panel  had  rated  favorably.  For 

example,  in  one  instance  it  became  evident  which  member 

$ 

of  a bidder’s  proposal  team  wrote  the  responses  to  certain 
sub-factors  which  received  high  ratings.  ETIP  was  then 
able  to  negotiate  for  this  person’s  involvement  in  the  pro- 
ject. 

V.3.4.2  Evaluation  of  technical  capability.  The  sec- 
ond issue  is  whether  PFE  provided  for  an  effective  evalua- 
tion of  technical  capability.  As  with  the  preceding  issue, 
this  primarily  examines  whether  ETIP  was  able  to  differen- 
tiate among  bidders  on  the  various  issues  which  were  inclu- 
ded as  sub-factors.  The  relationship  between  these  ratings 
and  anticipated  future  performance  is  also  discussed  in 
section  V. 3 • ^ • 3 • 

All  of  the  technical  factor  teams  were  successful  at 
agreeing  upon  unique  orderings  for  the  bidders  on  each  sub- 
factor. The  teams  were  able  to  discuss  the  information 
available  on  each  bidder  under  each  sub-factor  and  compare 
it  against  issues  in  question  and  the  criteria  for  evalua- 


tion. 
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Discussion  on  the  categories  of  the  Panel's  Role  in 
Evaluation  of  the  Proposals  (section  V.3.6),  Equity  of  Treat- 
ment Across  Bidders  (section  V.3.7),  and  Effective  Communi- 
cation (section  V.3.2)  indicate  some  of  the  elements  which 
affected  the  panel’s  ability  to  assign  scores.  Such  things 
as  the  range  of  panel  members’  initial  ratings,  the  biases 
of  various  panel  members,  and  the  amount  of  information 
available  under  each  sub-factor  heading  seemed  important. 

As  with  the  management  sub-factors,  the  clarifications 
sessions  proved  useful  for  gathering  additional  information 
where  the  panel  had  greater  uncertainty  in  the  scores  whic*h 
they  assigned. 

V.3.^.3  Anticipation  of  problems  known  to  exist  in 
the  project.  The  final  issue  under  this  category  asks  wheth- 
er PFE  anticipated  the  problems  which  were  known  to  exist 
in  doing  evaluations  and  in  developing  systems.  The  last 
two  issues  indicated  that  the  teams  were  able  to  differen- 
tiate among  bidders  and  assign  scores  on  both  management 
and  technical  sub-factors.  However,  these  scores  have  very 
little  meaning  unless  they  have  a positive  relationship 
to  subsequent  performance.  Before  the  RFP  was  issued,  a 
PAA  staff  member  indicated  that  he  felt  that  the  relation- 
ship would  be  inverse.  He  thought  that  any  bidder  who  would 
be  successful  in  responding  to  the  RFP  would  supply  the 
PAAs  with  useless  materials  once  under  contract  (section 
IV. 6) . 


In  order  to  do  an  adequate  job  of  responding  to  this 
issue,  the  subsequent  performance  of  the  successful 
bidders  would  have  to  be  assessed  and  compared  to  the  prob- 
lems which  arose  during  the  contracts  and  the  sub-factors 
which  were  included  in  the  RFP.  Such  a detailed  analysis 
of  the  post  award  time  period  is  not  within  the  scope  of 
this  dissertation.  However,  some  of  this  information  is 
contained  in  a report  by  Garrity  (1980).  Without  doing 
a detailed  analysis,  the  most  that  can  be  said  about  the 
relationship  between  the  evaluation  and  subsequent  perfor- 
mance is  that  it  seemed  mixed. 

In  analyzing  how  well  PFE  anticipated  future  perfor- 
mance, there  are  two  variables  which  seem  important.  One 
is  whether  ETIP  knew  about  all  of  the  critical  issues  in 
advance  and  the  second  is  whether  there  was  a way  to  m.easure 
accurately  the  bidders  acceptability  on  each  issue. 

On  the  first  variable  of  whether  the  critical  issues 
were  known,  ETIP  seemed  fairly  strong.  One  measure  of  this 
is  that  on  subsequent  procurements  for  similar  contracts 
in  support  of  ETIP's  regulatory  work  very  few  new  sub-factors 
were  added. 

There  were  more  substantial  problems  on  the  second 

$ 

variable  of  measurability.  In  large  part  this  was  a problem 
of  trying  -to  predict  bidder’s  ability  on  sub-factors  into 
the  future.  The  author  believes  that  ETIP’s  measurements 
of  the  bidders  against  the  criteria  were  fairly  accurate 
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at  the  time  that  they  were  made.  However,  circumstances 
which  occurred  over  time  seemed  to  change  the  bidders  posi- 
tion on  some  of  the  sub-factors.  This  was  particularly 
problematic  on  sub-factor  1.2,  Importance  of  Program  to 
the  Bidder.  The  bidders’  acceptability  on  this  sub-factor 
seemed  to  change  over  time.  Since  this  sub-factor  affects 
the  resources  which  are  devoted  to  capabilities  on  other 
sub-factors,  the  acceptability  on  other  sub-factors  also 
varied  over  time. 

The  inability  to  make  measurements  which  predict  cap- 
ability into  the  future  seemed  to  be  one  of  the  failings 
of  the  source  selection.  However,  it  is  difficult  to  attri- 
bute this  to  the  PFE  process  since  it  is  doubtful  that  any 
other  source  selection  techniques  are  effective  at  this 
either.  It  is  one  of  the  risks  of  awarding  a contract. 

V.3.5  Use  of  Experts  in  the  Evaluation  of  Proposals 

The  category  of  the  Use  of  Experts  in  the  Evaluation 
of  Proposals  is  concerned  with  the  issues  of  both  how  to 
obtain  the  participation  of  experts  and  how  to  make  use 
of  the  information  provided  by  experts. 

V.3.5. 1 Obtaining  participation  by  experts.  The  issue 
of  obtaining  the  participation  by  experts  includes  identify- 

t 

ing  them,  convincing  them  to  participate,  and  having  a sche- 
dule which  allows  them  to  participate.  This  is  identified 
as  a problem  in  the  literature  on  systems  procurement  (e.g., 
Carnes,  1976).  ETIP  was  quite  successful  on  this  issue. 


Experts  were  identified 


by  the  COTR  and  the  author 


throu 

gh  discussion  ' 

with 

other  ETIP 

and 

PAA 

staff 

and 

f rom 

their 

own  knowledge 

of 

relevant  people 

who 

might 

par 

t ic  i- 

pate . 

The  nineteen 

people  outside 

of  ETIP 

who  we 

r 6 

ident i 

f ied 

included  those 

who 

were : 

o either  part  of  or  closely  related  to  the  pro- 
curement experiments  in  the  PAAs, 

o thought  to  be  experts  on  some  of  the  specific 
sub-factors  being  evaluated,  or 

o familiar  with  some  of  the  likely  bidders. 

Some  of  these  people  were  considered  because  in  addi- 
tion to  their  expert  information  they: 

0 could  lend  credibility  to  the  project  with 
the  NBS  program  office,  or 

0 were  responsible  for  managing  other  government 
evaluation  programs  and  might  have  had  an  inter- 
est in  learning  of  the  PFE  technique  as  a method 
for  procuring  contractor  support. 

In  convincing  experts  to  participate,  the  project  and 
evaluation  process  were  explained  to  them  and  the  ability 
to  limit  participation  to  specific  sub-factors  was  stressed 
Of  the  nineteen  outside  people  who  were  invited  to  join 
the  team,  nine  agreed  to  participate  and  two  agreed  to  send 
representatives.  Those  who  declined  generally  indicated 
prior  commitments  during  the  week  for  which  the  evaluation 
was  scheduled.  It  was  easiest  to  get  people  in  other  pro- 
grams in  NBS  and  in  related  programs  in  the  National  Scienc 
Foundation.  People  who  were  in  evaluation  programs  which 


were  not  substantively  related  to  ETIP’s  work  most  frequent- 
ly indicated  that  they  were  busy. 

Scheduling  was,  for  the  most  part,  quite  successful 
but  did  present  some  problems.  The  schedule  for  the  times 
when  specific  factor  teams  would  meet  was  not  drawn  up  until 
the  panel  met  to  receive  the  proposals.  The  best  schedule 
which  could  be  drawn  up  resulted  in  several  people  being 
unable  to  participate  due  to  conflicts  at  different  points 
during  the  evaluation  week.  In  all,  eight  people  from  out- 
side of  ETIP  participated. 

V.3.5.2  Use  of  expert  information.  The  panel  members 
were  mixed  in  their  reaction  to  the  use  of  outsiders  in 
the  process.  In  response  to  the  question  on  the  strongest 
aspect  of  the  process,  one  of  the  ETIP  panel  members  felt 
that  being  able  to  allow  experts  to  limit  their  contribution 
was  the  greatest  strength  of  the  process  (see  Figure  V.30). 
However,  three  outside  participants  had  doubts  about  the 
usefulness  their  contributions  to  the  evaluation.  Two  of 
their  responses  suggest  that  experts  should  at  least  have 
some  knowledge  of,  and  relationship  to,  the  projects. 

Response  to  Panel  Question  5 on  the  Strongest  Aspect  of 
the  Process 

o It  enabled  people  with  particular  expertise  to  limit 
their  contribution  to  the  relevant  areas  (PM2). 
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Responses  to  Panel  Question  6 on  the  Weakest  Aspect  of  the 

Process 

o Being  present  at  only  some  of  the  factor  team  meetings 
limited  the  contribution  a person  could  make  and  stim- 
ulation they  could  receive  which  might  prompt  further 
useful  responses.  There  was  clearly  a trade-off  with 
the  burdens  of  having  everyone  participate  in  every- 
thing. These  tradeoffs  are  difficult  to  make  (PM5). 

o You  have  to  be  very  careful  of  the  people  that  you 
pick  for  it.  I may  not  have  had  enough  knowledge  of 
what  the  actual  program  was  and  a good  enough  under- 
standing of  just  what  was  wanted  in  the  contract  (PMl6). 

0 There  were  too  many  people  with  too  little  understand- 
ing of  the  total  project.  If  you  had  fewer  people 
and  limited  it  to  ETIP  and  GSA  staff,  you  would  have 
gotten  the  same  or  better  results.  It  took  us  a long 
time  to  get  an  idea  of  what  you  wanted  (PM6). 

FIGURE  V.30  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  EXPERT  PARTICIPATION 


V.3*6  The  Panel’s  Role  in  Evaluation  of  the  Proposals 

The  category  of  the  Panel’s  Role  in  Evaluation  of  the 
Proposals  is  concerned  with  the  effectiveness  of  the  factor 
teams  in  reviewing  the  information  available  about  the  bid- 
ders and  reaching  agreements  on  ratings  for  all  of  the  sub- 
f actors . 

Six  issue  discussions  are  included  under  this  category. 
They  are: 

1. a)  Did  panel  members  who  agreed  to  participate 

actually  show  up? 

How  difficult  was  it  to  schedule  meetings? 

b)  Were  panel  members  able  to  arrive  at  initial 
ratings  prior  to  factor  team  meetings? 

2.  Was  the  panel  able  to  arrive  at  final  scores 
without  voting? 

Was  the  panel  able  to  deal  effectively 
with  strong  individual  disagreements? 


3.  a)  Did  the  panel  members  realize  how  their  indi- 

vidual positions  affect  the  final  score, 
i.e.,  do  they  see  other  panel  members  modify- 
ing their  positions  in  response  to  the  discus- 
sion? 

Do  panel  members  understand  how  the  final 
score  is  reached? 

b)  Do  the  final  scores  differ  significantly 
from  the  means  of  the  initial  scores  of  the 
individual  panel  members? 

c)  Were  the  panels^  ratings  and  recommendations 
used  after  the  panel  meetings  were  concluded? 

4.  What  impact  did  those  with  extreme  scores 

or  greater  variability  in  their  ratings  have 
on  the  final  scores? 

5.  How  long  did  it  take  to  reach  agreement  on 
factor  scores  after  the  sub-factor  scores 
were  agreed  to? 

6.  Were  "junior”  panel  members  able  to  affect 
the  scores  if  their  bosses  were  in  the  room 
and  their  initial  scores  differed? 

V.3.6.1  Meeting  attendance  and  ability  to  determine 
initial  ratings.  The  first  issues  related  to  the  panel 
process  is  whether  people  who  agreed  to  participate  actually 
showed  up  at  the  factor  team  meetings  and  whether  they  were 
able  to  arrive  at  initial  ratings  prior  to  the  start  of 
these  meetings. 

Participation  in  the  evaluation  by  those  who  agreed 
to  be  panel  members  was  good.  Each  of  the  .two  contracts 
had  12  people  on  the  panel,  with  some  overlap  in  membership. 
Of  the  people  who  received  copies  of  the  proposals  at  the 
April  16  proposal  distribution,  two  were  unable  to  attend 
due  to  other  activities  which  had  higher  priority  for  them. 


With  one  exception,  the  panel  members  were  able  to 
arrive  at  initial  ratings  of  bidders  prior  to  factor  team 
meetings.  The  one  exception  was  a person  who  felt  that 
rating  bidders  on  the  sub-factors  was  evaluating  the  issues 
out  of  context.  He  came  to  his  factor  team’s  meeting  to 
indicate  his  willingness  to  discuss  the  proposals  as  a whole 
but  he  declined  to  participate  in  the  evaluation  process 
(see  Section  IV. 10. 10). 

The  initial  ratings  for  each  sub-factor  can  be  found 
in  the  appendix.  In  several  cases  one  of  the  panel  members 
had  a range  of  uncertainty  with  respect  to  one  or  more  bid- 
ders. This  panel  member  placed  the  bidders  initials  at 
two  different  scores  and  indicated  to  the  panel  that  he 
felt  that  the  bidder  was  somewhere  in  the  range  between 
the  scores.  In  the  cases  where  this  occurred,  ohe  average 
value  was  used  for  purposes  of  plotting  and  statistical 
computations . 

V.3.6.2  Reaching  agreement.  One  of  the  major  issues 
in  controversy  prior  to  the  beginning  of  the  evaluation 
was  whether  the  panel  would  be  able  to  arrive  at  final  scores 
without  voting.  This  was  particularly  important  because 
of  the  large  number  of  places  where  agreement  was  needed, 
i.e.,  on  33  sub-factors  for  each  of  two  contracts  and  on 
combining  the  sub-factor  scores  into  factor  scores.  It 
was  known  that  some  of  the  panel  members  held  divergent 
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views  about  the  objectives  of  the  projects  and  were  likely 
to  have  strong  disagreements  during  the  evaluation. 

In  all  cases  the  panel  was  able  to  arrive  at  final 
scores  without  voting.  In  cases  where  strong  disagreements 
persisted  after  team  members  presented  the  information  on 
which  they  based  their  ratings,  the  teams  used  two  mechan- 
isms to  facilitate  agreement.  The  method  most  commonly 
used  was 'to  assign  the  score  which  most  of  the  team  was 
tending  toward  and  indicate  that  the  issue  needed  further 
investigation  during  any  subsequent  clarifications.  The 
other  method,  which  was  used  twice,  was  to  post  both  scores 
on  the  board  and  proceed  with  discussion  on  other  sub-factors 
within  the  factor,  leaving  the  decision  on  the  sub-factor 
in  question  until  more  discussion  had  occurred  on  related 
issues.  In  no  case  did  the  panel  have  to  resort  to  the 
mechanism  available  in  PFE  of  having  a dissenting  team  member 
enter  a signed  exception  into  the  record. 

The  way  the  rating  process  generally  worked  was  that 
following  the  posting  of  initial  sub-factor  ratings  a bidder 
was  selected  for  discussion  and  a panel  member  with  either 
the  highest  or  lowest  score  was  asked  to  present  the  reasons 
for  his  rating.  After  he  finished,  someone  with  the  oppos- 
ing opinion  would  be  asked  to  talk.  Other  panel  members 
entered  into  the  discussion  indicating  what  evidence  they 
were  using  in  either  agreeing  or  disagreeing  with  the  posi- 
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tions  presented  or  adding  an  additional  perspective, 
the  discussion  continued,  team  members  would  either  rein- 
force or  modify  their  initial  perspective.  If  most  of  the 
team  was  tending  toward  one  score  but  one  or  two  members 
still  felt  that  a different  score  was  appropriate,  the  panel 
would  focus  on  the  minority  views.  People  would  look  for 
information  either  opposed  to  or  in  support  of  the  minority 
opinion.  In  some  cases  related  information  on  the  bidder's 
understanding,  approach  or  capability  for  dealing  with  the 
sub-factor  would  be  discovered  under  another  sub-factor 
which  had  not  been  cross  referenced.  In  other  cases  infor- 
mation from  a panel  member  who  was  familiar  with  the  bid- 
der's past  performance  on  the  issues  would  be  presented. 
Discussion  would  continue  until  the  panel  agreed  on  a score. 

In  some  cases  the  score  would  be  closer  to  that  held 
by  the  minority  position  than  that  of  the  m.ajority.  For 
example  Figure  V.31  shows  the  initial  and  final  ratings 
for  sub-factor  VI. 2 on  the  agency  impact  contract  evalua- 
tion. There  are  two  examples  in  this  sub-factor  of  a minor- 
ity opinion  convincing  the  majority.  Evaluator  205  was 
able  to  convince  the  other  team  members  that  Bidder  204 's 
proposal  was  unacceptable  on  this  sub-factor  without  signif- 
icant revisions  (a  score  of  four).  Likewise  Evaluator  211 
was  able  to  convince  the  team  that  Bidder  202  was  acceptable 
as  is  (a  score  of  seven). 
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FIGURE  V.31  EXAMPLE  OF  MINORITY  OPINIONS  PREVAILING 


The  time  taken  to  get  through  the  factor  meetings  var- 
ied. The  shortest  team  meeting  was  forty  minutes  and  the 
longest  was  two  hours  and  thirty-five  minutes.  The  average 
time  was  about  an  hour  and  a half  (see  Figure  '1.32). 


Agency  Impact 

Commercial  Impact 

Factor 

0 

I 

74  minutes 

46  minutes 

II 

104 

58 

III 

1 13 

105 

IV 

74  • 

40 

V 

155 

76 

VI 

120 

91 

VII 

127 

115 

X = 1 10 

X = '^6 

SD  = 29 

SD  = 29 

R = 81 

X = 93 

SD  = 33 

R = 75 

R = 1 15 

These  numbers 

are  estimates  of  the 

meeting  times  which  we 

computed  from 

running  the  tapes  of 

the  meetings.  They  do 

not  include  times  for  breaks  which 

were  taken  during  seve 

of  the  meetings. 

FIGURE  V.32  TIME  REQUIRED  FOR  THE  FACTOR  TEAM  MEETINGS 


Figure  V.33  presents  responses  to  the  open  ended  ques- 
tions which  to  some  degree  relate  to  reaching  agreement 
on  sub-factor  scores.  Several  panel  members  felt  that  the 
deliberation  and  reaching  agreement  on  the  sub-factors  was 
either  the  strongest  aspect  of  the  process  or  the  one  which 
they  remember  best.  Two  others  felt  that  the  requirement 
to  make  judgments  or  reach  consensus  on  each  sub-factor 


Response  to  Panel  Question  5 on  the  Strongest  Aspect  of 
the  Process 


o The  detail  of  the  sub-factors.  More  thought  was  given 
to  what  you  really  wanted  and  there  was  quite  a bit  of 
deliberation  on  the  sub-factors  (PM7). 

t 

o The  group  discussion.  Everyone  brought  relevant  informa- 
tion and  people  had  different  impressions  of  what  the 
bidders  meant  and  you  were  able  to  agree  on  what  they 
meant  (PM4 ) . 

o The  combination  of  formal  disaggregated  review  and  discus- 
sion periods  presented  and,  in  some  cases  altered,  in 
light  of  other  reviewers’  arguments,  information,  and 
perspective  (PM3). 

0 The  group  meeting  and  discussion  in  coming  up  with  final 
factor  scores  was  by  far  the  strongest  part.  The  team 
effort. 

Response  to  Panel  Question  6 on  the  Weakest  Aspect  of  the 

Process 


o The  sub-factors.  Making  an  individual  judgment  on  each 
one  got  too  complicated.  You  could  have  let  them  address 
the  factors  and  use  the  sub-factors  as  things  to  think 
about  (PM9). 

o The  insistence  on  a consensus  and  the  unnecessary  complex- 
ity (PM2). 

Response  to  Panel  Question  10  on  Any  Other  Aspects  of  the 

Process  Which  You  Wish  to  Get  Out 

o The  one  thing  I remember  quite  well  is  that  every  single 
person  initially  had  different  rankings  and  they  were 
able  to  arrive  at  consensus.  It  was  either  that,  or  peopl 
were  very  tired  of  the  lengthy  process  (PM15). 


FIGURE  V.33  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  REACHING  AGREEMENT 


was  the  weakest  aspect  of  the  process.  Other  issues  in 
this  category  which  are  closely  related  to  this  one  are 
those  on:  1)  whether  panel  members  realize  their  impact 

on  the  team's  scores  (section  V. 3*6.3))  2)  the  impact  of 
those  with  greater  variability  on  the  final  scores  (section 
V.3.6.4),  and  3)  the  time  required  to  reach  agreement  on 
the  weighting  of  sub-factors  within  a factor  (section 
V.3.6.5).  The  issue  dealing  with  the  bias  of  panel  members 
is  also  closely  related  to  this  one  (section  V. 3*7.1). 

V.3.6.3  Motivation  of  panel.  An  issue  which  is  impor- 
tant to  the  motivation  of  panel  members  is  whether  they 
understand  how  their  individual  contributions  affect  the 
results  of  the  process.  As  part  of  this,  they  must  also 
feel  that  these  results  will  be  used  and  that  they  are  not 
just  going  through  the  motions.  This  is  an  issue  which 
appears,  to  a small  degree,  in  the  group  process  literature 
(e.g.,  Bradford,  et  al.,  1953). 

The  mechanics  of  PFE  ensured  that  panel  members  were 
able  to  realize  the  impact  of  their  contributions  to  the 
scores.  This  occurred  through  the  requirements  for  all 
team  members  to  put  their  initial  ratings  of  bidders  into 
the  formal  records  of  the  process,  to  reach  agreement  on 
the  sub-factor  scores,  and  to  sign  the  final  factor  rating 
sheets.  Through  the  discussion  of  each  bidder  the  panel 
was  able  to  see  others  modify  their  initial  positions  as 
information  was  put  onto  the  table  and  evaluated.  By  not 


allowing  voting,  the  panel  had  to  confront  and  resolve  their 
differences.  A dissenting  opinion  could  not  be  ignored. 

This  was  true  for  both  the  individual  sub-factor  scores 
and  the  final  factor  scores.  The  contrast  between  this 
and  the  methods  used  by  some  other  proposal  evaluation  tech- 
niques was  highlighted  to  the  author  by  a colleague  who 
had  returned  from  participating  on  another  proposal  evalua- 
tion panel.  He  indicated  that  in  contrast  with  his  experi- 
ence on  a PFE  panel,  he  had  a very  fuzzy  understanding  of 
how  his  discussion  affected  the  panel's  scores.  He  also 
had  no  idea  how  these  scores  would  be  combined  to  reach 
a decision  on  which  bidder  to  negotiate  with. 

Another  way  the  panel  members  realized  their  impacts 
is  that  each  team  owned  the  process  for  coming  to  agreement. 
They  could  decide  upon  the  order  of  discussion  of  the  sub- 
factors and  the  methods  for  reaching  agreement.  In  most 
cases  the  team  chose  to  discuss  each  sub-factor  sequentially 
and  reach  agreement  prior  to  moving  on.  Section  IV. 10. 12 
described  the  one  meeting  where  the  team  chose  to  discuss 
the  factor  as  a whole  prior  to  rating  the  individual  sub- 
factors. Both  methods  worked  well. 

Several  of  the  responses  listed  in  Figure  V.32  pertain 
to  the  issue  of  satisfaction  with  the  group  process.  Five 
of  the  panel  respondents  chose  to  mention  their  satisfaction 
with  the  group  decision  process  under  the  open  ended  ques- 
tions. Two  mentioned  dissatisfaction. 


The  other  part  of  this  issue  is  whether  the  panel's 
inputs  were  used  following  the  final  team  meeting  and  wheth- 
er the  panel  members  had  any  indication  of  how  they  would 


be  used.  The  PFE  process  was  very  effective  at  incorporat- 
ing the  panel's  recommendations  into  clarification  sessions 
with  the  bidders  who  were  in  the  competitive  range.  The 
COTR's  report  to  the  contract  negotiator  was  written  by 
using  the  actual  sub-factor  scores  to  describe  the  areas 


where  bidders  were  found  to  have 

problems  requiring  clarifi- 

cation,  negotiation,  or  revision 

(see  section  IV. 11).  The 

competitive  range  was  determined 

by  examining  both  the  over- 

all  scores  and  the  distribution  of  individual  sub-factor 
scores . 

During  the  panel  meetings,  areas  of  uncertainty  were 
noted  in  the  formal  record  of  the  process  when  the  panel 
indicated  that  clarifications  were  required.  Since  the 
process  for  noting  issues  for  clarification  was  mentioned 


quite  frequently,  the  panel  knew 

that  these  notes  would 

go  to  the  contracting  office  for 

follow-up.  The  questions 

for  the  oral  clarifications  were 

based  upon  these  notes 

and  the  sub-factors  on  which  the 
below  a seven. 

teams  had  rated  the  bidders 

Some  additional  discussion  related  to  this  issue  is 
included  in  section  V.3.11  which  described  the  reaction 
of  the  contracting  office  to  a suggestion  for  a committee 
to  review  the  panels'  recommendations. 


V.3.6.4  Dealing  with  extreme  scores  or  great  variabil- 
ity. One  of  the  difficult  problems  to  deal  with  in  proposal 
evaluation  is  how  to  agree  upon  a final  score  when  individ- 
uals are  using  different  standards  for  their  evaluation. 

This  includes  the  question  of  what  impact  those  team  members 
with  extreme  initial  scores  or  greater  variability  in  their 
ratings  had  on  the  final  scores.  Figure  V.34  illustrates 
the  different  types  of  distributions  in  scores  which  are 
possible.  Graph  A is  an  example  of  the  range  of  scores 
of  a team  member  with  a great  deal  of  variability  in  his 
ratings.  He  finds  that  some  bidders  are  excellent  and  that 
some  bidders  are  awful  on  the  sub-factor  issues.  Graph 
B is  an  example  of  the  range  of  a team  member  with  a moder- 
ate amount  of  variability  in  his  ratings.  He  seldom  assigns 
extremely  high  or  extremely  low  scores.  Graph  C is  an  ex- 
ample of  someone  with  a small  amount  of  variability  in  his 
ratings.  He  finds  that  most  or  all  bidders  are  acceptable 
on  the  sub-factor  issues.  Graph  D is  an  example  of  someone 
with  a small  amount  of  variability  in  his  ratings.  To  him 
most  or  all  bidders  are  unacceptable  on  the  sub-factor  is- 
sues. All  of  these  situations  arose  during  the  team  meet- 
ings . 

Through  discussion  the  team  members  were  able  to  agree 
on  scores.  Those  who  took  extreme  positions  were  required 
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A B C D 

v.34  examples  of  different  types  of  variability 

USED  IN  RATING  THE  BIDDERS 


to  justify  their  scores  against  the  scale.  For  example, 
if  someone  rated  a bidder  as  a four,  in  contrast  to  higher 
ratings  of  others,  the  other  members  pressed  him  to  indicate 
why  the  bidder  was  unacceptable  without  significant  revi- 
sions and  to  describe  what  specific  types  of  revisions  he 
felt  were  necessary.  Referring  to  the  words  on  the  scale 
and  making  relative  comparisons  to  ratings  assigned  to  other 
bidders  seemed  to  be  very  helpful  in  achieving  agreement. 

Figure  V.35  presents  a good  example  of  differences 
in  variability  among  evaluators  and  shows  the  factor  team’s 
ability  to  deal  with  it.  Note  that  Evaluator  205  and  208 
made  considerable  changes  in  their  ratings  after  the  team’s 
discussions  comparing  standards  and  other  information  re- 
lated to  the  proposals. 

Despite  the  teams’  ability  to  reach  agreement,  one 
panel  member  still  felt  that  insufficient  attention  was 
given  to  comparing  standards  on  the  meaning  of  sub-factors 
during  the  evaluation  (see  Figure  V.36). 

V.3.6.5  Time  required  to  agree  on  sub-factor  weights. 
One  of  the  major,  concerns  prior  to  the  evaluation  was  that 
the  process  of  arriving  at  factor  scores  once  the  sub-factor 
scores  were  known  would  be  extremely ^time  consuming  (see 
section  IV. 8).  It  was  feared  that  it  would  be  difficult 
to 'determine  the  relative  weightings  of  the  sub-factors 
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Response  to  Panel  Question  6 on  Weakest  Aspect  of  the  Pro- 
cess 

o Needs  to  have  more  spelled  out  during  the  evaluation  about 
what  the  sub-factors  mean  to  assure  that  everyone  is  using 
the  same  standards  (PM1). 

FIGURE  V.36  RESPONSE  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATES 
TO  EXTREME  SCORES  OR  GREATER  VARIABILITY 

within  each  factor  if  this  decision  was  left  up  to  each 

0 

team.  It  was  also  thought  that  doing  this  could  create 
impropriety  if  the  weighting  were  not  agreed  to  until  after 
the  sub-factor  scores  were  known,  or  that  the  appearance 
of  impropriety  would  be  present  even  if  the  sub-factor  rat- 
ings were  agreed  to  at  the  start  of  the  factor  team  meeting. 

This  issue  did  not  present  any  time  delay  problems 
during  the  evaluation.  In  all  cases,  the  weightings  were 
agreed  to  after  the  sub-factors  were  rated  and  in  no  case 
did  the  scoring  take  more,  than  six  minutes.  The  process 
for  arriving  at  the  factor  scores  was  for  the  most  part 
one  of  taking  an  eyeball  average  of  the  sub-factor  scores. 

If  the  bidders  fell  between  two  factor  scores,  there  was 
discussion  of  whether  some  of  the  factors  should  be  weighted 
more  heavily  than  others.  For  example,  one  panel  member 
felt  that  sub-factors  dealing  with  Phases  II  and  III  of 
the  project  should  be  given  less  weight.  Where  the  panel 
felt  there  was  justification  for  weighting  a sub-factor 
more  heavily,  that  was  taken  into  account  in  the  final 


scores . 


If  not,  a visual  average  of  the  scores  would  be  assigned. 
Figures  V.37A  and  V.37B  display  the  differences  between 
the  numerical  averages  of  the  sub-factor  scores  and  the 
agreed  upon  factor  scores.  The  differences  are  generally 
fairly  small. 
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The  question  of  whether  the  process  of  agreeing  on 
the  factor  scores  presented  the  appearance  of  impropriety 
is  investigated  in  section 

V.3.6.6  Input  by  .junior  panel  members.  The  final 
issue  to  be  considered  in  this  category  is  whether  "junior" 
panel  members  were  able  to  affect  the  panel’s  score  if  their 
bosses  were  in  the  room  and  their  initial  scores  differed 
(e.g.,  see  Torrance,  1957). 

It  is  unclear  how  effective  PFE  was  in  dealing  with 
this  issue.  There  were  several  cases  where  junior  panel 
members  and  their  bosses  differed  and  the  panel  took  the 
position  of  the  junior  panel  member.  However,  frequently 
the  junior  member  was  already  in  agreement  with  the  majority 
of  the  panel. 

» 

PFE  has  a mechanism  for  dealing  with  the  tendency  of 
junior  panel  members  to  agree  with  their  bosses.  3y  having 
the  initial  ratings  posted  and  put  into  the  formal  record, 
an  incentive  is  created  for  each  panel  member  to  make  both 
the  reasons  for  his  initial  score  and  his  changes  in  the 
score  known  to  the  others.  This  may  cause  the  junior  panel 
member  to  defend  his  position  more  actively.  However,  the 
author  feels  that  the  evidence  available  from  the  data  is 
inconclusive  on  this  issue. 

V.3.7  Equity  of  Treatment  Across  Bidders 
The  category  of  Equity  of  Treatment  Across  Bidders 
is  concerned  with  whether  the  panel  applied  any  undue  differ- 


ential  evaluation  for  or  against  any  of  the  bidders.  Of 
Goncern  are  issues  of  how  bias  was  dealt  with,  whether  the 
criteria  were  applied  consistently,  and  whether  the  lack 
of  restriction  on  admissible  information  (including  knowl- 
edge of  the  identity  of  the  bidders)  created  any  inequity. 

Three  issue  discussions  are  included  under  this  cate- 
gory. They  are: 

'1.  Did  the  panel  deal  effectively  with  the  biases 
of  its  members? 

2.  Did  the  criteria  used  in  evaluation  provide 
for  a consistent  evaluation  across  proposals? 

Were  the  criteria  applied  consistently 
across  proposals? 

3.  a)  Did  the  lack  of  restriction  on  admissible 

information  lead  to  any  inequity  in  the  pro- 
cess? 

b)  Did  evaluation  of  the  proposals  while  knowing 
the  identity  of  the  bidders  create  any  prob- 
lems - biases  in  the  final  ratings,  protests? 

V.3.7.1  Treatment  of  bias.  The  issue  of  bias  is  not 
whether  or  not  there  was  bias  among  the  panel  members. 

There  was.  It  is  a matter  of  whether  the  panel  incorporated 
this  bias  into  the  evaluation  process  in  a manner  which 
did  not  improperly  advantage  or  disadvantage  any  of  the 
bidders . 

PFE  provided  a process  for  revealing  and  exploring 
the  biases  of  its  panel  members.  First,  factor  team  members 
were  required  to  sign  and  turn  in  their  initial  independent 
ratings  of  the  bidders  prior  to  discussion  on  the  sub- 
factors . 
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Those  ratings  became  part  of  the  record  of  the  process. 

Next,  panel  members  were  required  to  discuss  the  basis  for 
their  ratings.  This  was  true  for  low  as  well  as  high  rat- 
ings. Those  team  members  rating  each  sub-factor  were  re- 
quired to  continue  the  discussion  until  they  reached  agree- 
ment on  a score.  They  could  not  resolve  their  di sagreements 
by  voting  or  merely  averaging  their  scores.  They  were  re- 
quired to  explore  the  bases  for  their  differences  and  dis- 
cuss the  evidence  they  were  using.  A taped  record  was  made 
of  this  discussion.  As  others  on  the  factor  team  would 
participate  in  the  arguments,  the  team  members  would  gen- 
erally be  drawn  toward  a position.  If  one  or  more  members 
of  the  team  persisted  in  having  a strong  disagreement  with 
the  score  which  mobt  of  the  team  wanted,  there  was  a mechan- 
ism for  assigning  this  score  and  recording  the  signed  dis- 
senting opinion.  This  mechanism  was  never  used.  The  closest 
the  teams  came  to  this  was  assigning  a score  and  noting 
in  the  record  that  clarification  was  required  on  the  points 
of  uncertainty.  The  signed  rating  sheets  and  the  tape  record- 
ings created  a record  of  all  biases.  The  process  of  requir- 
ing team  members  to  reach  agreement  required  an  evaluation 
of  the  merits  of  these  biases. 

One  special  situation  of  dealing  with  bias  is  the  case 
of  being  biased  for  or  against  specific  bidders  because 
of  prior  experience  with  them  as  contractors.  For  example. 


CO 


with  one  of  the  bidders  who  had  been  an  ETIP  contractor, 
one  panel  member  claimed  that  there  was  no  reason  to  rate 
this  bidder  since  he  had  worked  with  him  and  knew  that  he 
was  totally  unacceptable.  The  panel  understood  how  this 
person  felt  about  this  bidder  but  realized  it  was  still 
necessary  to  rate  this  bidder  on  his  merits  on  each  separate 
sub-factor . 

Some  calculations  were  run  on  the  data  in  order  to 
examine  the  biases  and  the  extent  of  influence  of  the  vari- 
ous members  of  the  evaluation  panel.  As  a first  step,  each 
panel  member’s  ratings  of  the  bidders  were  changed  to  rank- 
ings in  order  to  eliminate  the  problems  of  differential 

n 

variability  among  the  raters  (see  section  V.3.6.4).'  Pat- 
terns of  orderings  of  the  bidders  seemed  more  important 
in  computing  bias  that  the  actual  numbers  assigned.  To 
obtain  a measurement  of  each  members  bias,  the  difference 
between  that  member’s  rating  of  each  bidder  on  a given  sub- 
factor and  the  average  rating  of  other  panel  members  evaluat- 
ing the  same  bidders  on  the  sub-factor  was  computed.  A 
summation  of  these  differences  was  then  computed  across 
all  sub-factors  which  the  panel  member  participated  on. 

I.e.,  bias  of  panel  member  I for  bidder  J ,is  computed  as: 

7 

A correction  factor  was  included  for  panel  members 
who  failed  to  rate  all  of  the  bidders  on  a sub-factor. 

Where  there  were  six  bidders  and  the  panel  member  only  rated 
five  of  them  the  ordering  was  shifted  from  1,  2,  3>  4,  5 
to  1.5,  2.5,  3«5,  4.5,  5.5,  where  1 is  the  lowest  rating 
and  six  is  the  highest. 


250 


BIAS 


IJ 


RANK 


IJk 


k=sub-f actors 
on  which  I 
participated 
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i=all  other 
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M 


where , 

n = total  number  of  panel  members  rating  bidder  J on  sub- 
factor K,  and 

N = total  number  of  sub-factors  on  which  panel  member  1 
rated  bidder  J. 

The  assumption  underlying  this  computation  is  that  if  a 
panel  member  is  unbiased  toward  bidder  J,  relative  to  the 
other  panel  members,  BIAS^j  should  be  close  to  zero,  i.e., 
his  ranking  of  bidder  J should  be  higher  than  the  other 
panel  members’  rankings  about  as  often  as  it  is  lower. 

A large  positive  or  large  negative  value  for  BlASxj  would 
indicate  a relative  bias  for  or  against  bidder  J. 

The  next  step  was  to  compute  a measurement  of  each 
member's  influence  over  the  final  rankings  of  the  panel. 
This  was  done  by  determining  the  degree  to  which  the  direC' 
tion  of  the  difference  between  the  final  rankings  and  the 
average  initial  rankings  tended  to  follow  each  panel  mem- 
ber's rankings.  The  differences  were  computed  for  each 
sub-factor  as  follows: 
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PANEL  DIFFERENCE 


jk 


FINAL  RANK 


jk 


RANK 

1 

i=all  panel 
members  on 
sub-factor  k 


/(n-1) 

ijk 


where , 

n = total  number  of  panel  members  rating  bidder  J on  sub- 
factor K. 

These  numbers  were  then  compared  to  the  values  of  individual 
differences  of  each  panel  member,  I,  where: 


INDIVIDUAL  DIFFERENCEj^j^  = RANK^^j^  - 2 RANK  . ( n- 1 ) 

i=all  other 
panel  members 
on  sub-factor 
k 

A table  was  constructed  for  each  panel  member  where,  if 

the  sign  of  PANEL  DIFFERENCE  ..  was  the  same  as  the  sign 

J ^ 

of  INDIVIDUAL  DIFFERENCE.,  a " + " was  entered.  If  the  signs 
were  different  a was  entered.  If  both  were  equal  to 
zero  a "0”  was  entered.  Figures  V.38  through  V.58  display 
the  values  and  tables  for  the  four  most  active  participants 
on  the  agency  impact  contract  evaluation  and  the  three  most 
active  participants  on  the  commercial  impact  contract. 
Figure  V.59  displays  the  sum  of  ratios  of  +/-  for  each  of 
the  evaluators,  a measure  of  overall  influence. 

Evaluator  205 *s  difference  scores  show  both  a large 
number  of  extreme  rankings  and  overall  bias  for  bidders 
201  and  206  and  against  bidder  202.  Examination  of  the 
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FIGURE  V.59  RATIO  OF  PANEL  MOVEMENT  TOWARD/AWAY  FROM 
RANKINGS  OF  EACH  PANEL  MEMBER 


influence  indicators  show  that  he  was  unable  to  sway  the 
other  panel  members.  For  bidders  201  and  202  the  teams' 
final  rankings  tended  to  move  away  from  their  average  rank- 
ings in  a direction  opposite  evaluator  205 's  initial  scores 
For  bidder  206  they  moved  slightly  toward  205’s  initial 
scores.  On  the  average,  across  bidders,  the  panel  rankings 
tended  to  move  toward  bidder  205 's  rankings  44^  of  the  time 
For  bidders  201,  202,  and  206,  these  percentages  were  23!o, 
31^  and  62^  respectively. 

Evaluator  208 ' s scores  show  bias  toward  bidder  205 
and  against  bidder  206.  For  bidder  205  the  teams'  rankings 
moved  somewhat  toward  evaluator  208 's  while  for  bidder  206 
they  moved  slightly  away  from  208 's.  Across  sub-factors 
the  panel  moved  toward  bidder  208 ' s rankings  ^9%  of  the 
time.  For  bidders  205  and  206  the  percentages  were  7'^% 
and  26^. 

The  scores  of  evaluators  210  and  211  do  no  show  any 
strong  bias  for  or  against  any  of  the  bidders.  For  evalu- 
ator 210  the  percentage  of  +/-  across,  bidders  is  44“5.  For 
evaluator  211  it  is  56.6%. 

The  percentage  scores  from  the  agency  impact  contract 
data  indicate  that  panel  members  had  varying  degrees  of 
influence  over  the  final  rankings.  Comparison  of  the  dif- 
ference scores  shows  that  panel  members  who  were  biased 
toward  specific  bidders  were  not  able  to  sway  the  results. 


In  fact,  the  panel  often  tended  to  move  away  from  those 
who  were  biased. 

The  data  for  commercial  impact  is  more  difficult  to 
interpret.  The  computations  for  bias  and  influence  are 
sensitive  to  missing  data,  the  number  of  bidders,  and  the 

g 

degree  of  initial  agreement  in  the  rankings.  Since  there 
were  only  three  bidders,  since  evaluator  102  failed  to  rate 
bidder  103  on  any  of  the  nineteen  sub-factors  on  which  he 
was  a participant,  and  since  there  was  a high  degree  of 
initial  agreement  in  the  scores,  it  is  difficult  to  say 
much  about  this  data,  other  than  that  with  the  exception 
of  bidder  102,  there  does  not  seem  to  be  any  bias  in  it. 

Figure  V.60  presents  two  responses  to  open  ended  ques- 
tions which  relate  to  the  treatment  of  bias. 

Other  issues  related  to  this  one  include:  how  the 

panel  dealt  with  differences  in  uncertainty  among  bidders 
during  the  evaluation  (discussed  in  section  V.3.2.4);  and 
whether  evaluation  while  knowing  the  identity  of  bidders 
created  any  problems  (discussed  in  section  V.3.7.3). 


Where  a panel  member  rates  only  bidder  101  and  102 
the  rankings  come  out  1.5  for  the  lowest  and  2.5  for  the 
highest.  If  there  is  a great  deal  of  initial  agreement 
that  101  is  highest  (i.e.,  those  rating  all  three  bidders 
rank  him  3),  the  panel  member  will  always  appear  biased 
against  bidder  101.  If  the  team  agrees  that  bidder  101 
is  highest,  the  final  team  score  will  always  appear  to  move 
away  the  panel  members  with  missing  data  and  toward  the 
other  panel  members. 
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Response  From  Panel  Question  5 on  Strongest  Aspect  of  the 
Process 


o The  system  was  superb.  I wouldn’t  want  to  do  it  any 
other  way.  Its  strength  is  that  you  don’t  have  to 
pretend  that  you  have  no  biases.  It  brings  a very 
professional  way  to  channel  personal  opinion  and  judg- 
ment and  feeling  in  an  honest,  constructive  fashion. 

It  makes  subjective  objective  (PM12). 

Response  From  Bidder  Question  11  on  Weakest  Aspect  of  the 

Process 

o It  gave  people  at  ETIP  an  opportunity  to  try  to  take 
sub jective ‘ things  and  make  them  objective  in  the  sense 
that  they  were  using  the  same  rigor  as  a serious  qual- 
ified proposal  team  would  use  in  laying  out  how  they 
were  going  to  respond.  I’m  not  saying  it  worked  (3R4). 

FIGURE  V.60  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  THE  TREATMENT  OF  BIAS 


V.3.7.2  Consistency  of  evaluation  across  bidders. 

The  next  issue  concerns  the  criteria  for  rating  sub-factors 
and  whether  they  provided  a consistent  evaluation  across 
all  proposals.  When  broad  categories  of  evaluation  criteria 
are  used  and  the  evaluation  is  made  against  separate  designs 
proposed  by  each  bidder,  there  may  be  considerable  diffi- 
culty in  achieving  a comparable  basis  for  evaluation.  Each 
proposal  will  surface  different  issues  and  there  may  be 
little  basis  for  comparability.  For  example,  in  a prior 
ETIP  RFP  for  evaluation  services,  the  broad  categories  of 
Methodology  and  Qualifications  of  Offeror  were  used.  There 
were  five  sub-categories  within  the  first  category  and  two 
within  the  second.  The  RFP  indicated  that  the  panel  would 
complete  its  evaluation  on  the  Methodology  category  before 


279 


beginning  its  evaluation  on  the  Qualifications  category. 

In  an  evaluation  like  this,  the  panel  must  read  through 
the  methodology  section  and  decide  on  criteria  within  each 
of  the  sub-categories.  Since  issues  which  the  bidders  choose 
to  discuss  will  vary,  it  may  be  difficult  to  get  a strong 
basis  for  comparison.  Also  since  qualifications  are  evalu- 
ated later,  the  scores  on  methodology  are  awarded  with  no 
consideration  given  to  the  capability  for  carrying  out  .the 
approaches  proposed.  When  qualifications  are  considered, 
the  structure  provides  no  basis  for  relating  them  back  to 
the  specific  approach' proposed  by  each  bidder.  The  result 
may  be  that  the  qualifications  points  are  awarded  merely 
on  the  reputations  of  the  various  people  whose  resumes  are 

included  in  the  proposal.  Consideration  to  whether  they 

» 

have  a capability  to  work  together  as  a team  to  manage  and 
carry  out  the  approach  proposed  may  be  neglected. 

The  PFE  process  was  successful  at  dealing  with  this 
issue.  The  sub-factors  provided  the  categories  for  consis- 
tent response  by  bidders  and  the  three  criteria  of  under- 
standing, approach  and  present  or  potential  capability  pro- 
vided for  consistent  evaluation.  All  bidders  were  required 
to  not  only  demonstrate  methods  or  approaches  but  they  were 
required  to  link  the  evidence  for  capability  to  each  method 
or  approach.  Every  bidder  was  evaluated  against  the  same 


issues . 
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Sometimes  during  a team  meeting  a panel  member  would 
argue  for  a high  score  for  a bidder  based  upon  his  proposed 
approach  for  dealing  with  a sub-factor.  Another  panel  mem- 
ber might  agree  that  the  approach  looked  good  but  point 
out  that  he  could  find  no  evidence  that  the  bidder  was  cap- 
able of  carrying  it  out.  The  evidence  on  all  three  criteria 
would  be  examined  until  agreement  was  reached. 

V.3.7.3  Admissible  information.  The  final  issue  in 
this  category  of  equity  of  treatment  is  the  impact  of  the 
lack  of  restriction  on  admissible  information.  There  was 
a strong  concern  over  both  potential  inequities  and  appear- 
ances of  inequities  which  might  be  caused  by  a "non-blind” 
evaluation  of  proposals  (see  section  IV. 3). 

The  ability  to  use  all  available  information  was  a 
strength  of  the  evaluation.  There  is  some  inconclusiveness 
about  whether  any  inequities  were  created.  A question  with 
two  sub-items  was  included  in  the  panel  questionnaire  to 
obtain  reaction  to  this  issue  the  items  and  responses  to 
them  are  presented  in  Figures  V.6I  and  V.62.  The  panel 

Did  being  able  to  use  all  available  information  about  bid- 
ders in  the  evaluation  improve  the  accuracy  of  the  scores? 

Yes  - 15  No  - 1 • 

Remark  by  a person  answering  Yes 

o I don’t  really  remember.  My  feeling  is  that  this  is  gen- 
erally more  useful  (PM6). 


CO 


Remark  by  person  answering  No 


o I think  it's  better  not  to  know  who  the  bidders  are  (PM1). 

FIGURE  V.61  PANEL  MEMBER’S  RESPONSES  TO  QUESTION  4a  WHICH 
RELATES  TO  ADMISSIBLE  INFORMATION 


Did  being  able  to  use  all  available  information  about  bid- 
ders in  the  evaluation  introduce  unfair  bias  into  the  scores 

Yes  - 3 No  - 13 

Remarks  by  those  answering  Yes 

o For  those  that  you  know,  you  start  reading  into  the  pro- 
posals things  that  they  don’t  mean  or  haven’t  said  because 
you  are  biased  (PM1). 

o The  additional  information  improves  accuracy,  as  would 
any  relevant  increment  of  information.  However,  the  per- 
son supplying  the  information  often  exhibits  a bias  (i.e., 
it  is  easier  and  more  comfortable  to  consider  working 
with  people  you  know)  (PM2). 

o If  one  individual  knew  the  firm  and  had  problems  with 
it,  that  convinced  the  rest  of  the  group.  Whether  they 
really  did  poor  work,  or  he  just  thought  it  was,  I don’t 
know.  It  did  carry  a lot  of  weight.  If  only  one  of  the 
team  knew  of  the  company  the  team  tended  to  accept  it. 

If  more  than  one  knew  it  there  would  be  discussion  and 
I think  it  would  come  out  fair  (PM16). 

Remarks  by  those  answering  No 

o Obviously  there  is  a value  judgment  as  to  what  is  unfair. 
Information  on  past  performance  is  relevant  and  not  at 
all  unfair  (PM5). 

o If  a panelist  has  had  any  unusual  or  direct  experience 
you  ought  to  be  aware  of  that  so  you  can  interpret  any 
bias  (PM9). 

If 'a  bidder  never  completes  their  work  on  time  and  there- 
fore you  don’t  select  him,  I don’t  think  it’s  unfair  bias. 
I think  it’s  a smart  way  of  doing  business  (PM4). 


o 


2 


0 One  proposal  had  been  submitted  by  a contractor  who  had 
not  done  a good  job  for  ETIP  in  the  past.  The  parts  of 
the  proposal  that  I read  would  lead  you  to  believe  that 
he  had  done  a superlative  job.  The  fact  that  we  could 
discuss  his  past  performance  and  bring  that  to  bear,  I 
think,  was  a good  idea  (PM15). 

FIGURE  V.62  PANEL  MEMBERS’  RESPONSES  TO  QUESTION  4b  WHICH 
RELATES  TO  ADMISSIBLE  INFORMATION 

members  generally  felt  that  being  able  to  use  all  available 
information  about  bidders  in  the  evaluation: 

o improved  the  accuracy  of  the  scores  (Yes  -15; 

No  - 1 ) ; and 

o introduced  unfair  bias  into  the  scores  (Yes 
it  did  - 3;  No  it  did  not  - 13). 

The  remarks  by  the  three  people  who  answered  Yes  to 
question  4b  (see  Figure  V.62)  indicate  how  they  feel  unfair 
bias  may  have  entered  into  the  scores.  PM1  and  PM2  indicate 
why  they  feel  that  people  evaluating  proposals  may  be  biased 
but  do  not  indicate  how  this  bias-  may  have  affected  the 
final  sub-factor/factor  scores  (as  compared  to  the  panel 
members’  initial  scores).  PM3  indicates  that  if  only  one 
person  knew  of  a bidder’s  prior  performance,  this  tended 
to  carry  a lot  of  weight.  Possibly  too  much  weight. 

Remarks  by  some  of  the  people  who  answered  No  to  ques- 
tion 4b  indicate  why  they  feel  that  the  use  of  outside  infor- 
mation led  to  proper  use  of  justifiable  biases.  They  indi- 
cate that  outside  information  allows  the  panel  to  evaluate 
the  information  in  the  proposals  better  and  to  interpret 


CO 
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the  biases  of  panel  members.  The  panel  does  not  have  to 
rely  only  on  what  the  proposal  writers  say  (e.g.,  see  re- 
sponse of  PM15,  Figure  V.62). 

"Blind”  proposal  evaluation  may  frequently  not  be  "blind" 
at  all.  Panel  members  will  recognize  bidders  they  are  fa- 
miliar with  through  the  approaches  proposed,  writing  style 
or  the  way  the  proposals  are  bound.  In  this  ca^e  panel 
members  do  not  have  to  justify  their  biases  (since  they 
are  not  able  to  admit  that  they  know  whose  proposal,  they 
are  evaluating).  Also  the  panel  is  not  able  to  differen- 
tially score  a bidder  who  has  not  worked  for  the  program 
before  and  makes  an  error  of  interpretation  and  a bidder 
who  has  worked  for  the  program  before  and  makes  an  error 
because  he  is  unable  to  learn.  This  relates  to  the  discus- 
sion under  section  V.3.2.4  on  differences  in  uncertainty 
among  bidders.  This  issue  also  relates  to  the  issue  of 
treatment  of  bias  (section  V.3-7.1)  and  those  on  traceabil- 
ity of  the  process  (section  V.3.IO). 

V.3.8  Risk  Associated  with  the  Statement  of  Work 

The  next  category  of  issues  concerns  whether  ETIP  was 
able  to  accomplish  the  writing  of  a flexible  statement  of 
work  and  deal  with  the  added  risk  associated  with  this. 

It  covers  issues  of  writing  and  getting  the  Statement  of 
Work  accepted,  having  bidders  understand  it,  and  being  able 


to  make  the  assessment  of  bidders’  objectives  necessary/ 
to  assuming  the  added  risk  associated  with  it. 

The  three  specific  issues  covered  in  this  section  are: 

1.  Was  ETIP  able  to  write  a statement  of  work 
which  would  spell  out  deliverables  and  yet 
avoid  numerous  contract  changes  which  generally 
occur  in  situations  where  requirements,  pol- 
icies, and  the  state  of  the  art  are  changing? 

Did  ETIP  management  accept  it? 

Did  the  contracting  officer  accept  it? 

2.  Did  the  flexible  statement  of  work  scare  off 
bidders? 

3.  Was  ETIP  able  to  assess  the  com.mitment  of  the 
bidders'  top  management  to  devote  the  necessary 
resources  to  the  project? 

Was  ETIP  able  to  determine  whether  it  was 
within  the  longer  term  interest  of  manage- 
ment to  develop  the  desired  systems? 

V.3.8.1  Flexible  statement  of  work.  The  first  issue 
under  this  category  is  whether  ETIP  was  able  to  write  an 
acceptable  statement  of  work  which  would  spell  out  deliver- 
ables and  yet  avoid  numerous  contract  changes  which  gener- 
ally occur  in  situations  where  requirements,  policies,  and 
the  state-of-the-art  are  changing.  Acceptable  is  taken 
in  the  proximal  sense  of  whether  a flexible  statement  of 
work  would  be  approved  by  both  the  program  and  the  contract 
ing  office. 

STIP's  results  on  this  issue  were  mixed.  Thompson 

$ 

was  able  to  write  a short  flexible  statement  of  work  which 
followed  the  format  of  the  factors  and  would  not  require 
modifications  as  changes  occurred.  It  indicated  that  both 
progress  and  final  reports  would  be  expected  on  both  techni 


cal  and  management  aspects  of  the  project.  However,  it 
left  the  dates  and  specific  subject  matter  of  these  reports 
undefined.  ETIP  management  felt  that  the  statement  of  work 
needed  to  be  more  specific.  Therefore,  Thompson  also  inclu- 
ded another  version  of  the  same  statement  of  work  to  which 
he  added  illustrative  sub-items  which  included  specific 
activities  and  reports  and  dates  for  the  reports.  These 
items  were  intended  only  as  examples  to  the  prospective 
bidders  of  the  types  of  things  which  could  be  furnished 
under  the  statement  of  work.  This  second  illustrative  state- 
ment of  work  caused  problems  for  ETIP  and  the  successful 
bidders  during  the  early  parts  of  Phase  I (see  section 
V.3. 14) 

The  contracting  officer  had  no  problems  with  the  state- 
ment of  work.  He  had  previously  contracted  for  ships  for 
the  Maritime  Administration  and  recognized  the  usefulness 
of  having  flexibility  in  a complex,  uncertain  contract. 

V.3.8.2  Effect  of  statement  of  work  on  bid  decisions. 

An  issue  which  was  raised  by  some  of  the  ETIP  staff  and 
PAA  staff  was  that  the  flexible  statement  of  work  would 
scare  off  bidders.  The  point  was  that  they  would  not  under- 
stand what  was  expected  of  them  and  would  therefore  not 
bid. 

The  answer  to  whether  this  happened  is  unclear.  It 
is  quite  possible  that  some  of  the  bidders  were  confused 
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ETIP’s  results  here  were  reported  under  the  first  and 
third  issues  of  section  V.3.^.  In  sum,  the  management  fac- 
tor team  for  the  agency  impact  evaluation  system  contract 
found  it  was  not  able  to  differentiate  among  bidders.  The 
clarifications  sessions  enabled  ETIP  to  assess  differences 
among  bidders  in  terms  of  which  objectives  they  would  devote 
resources  to  and  to  assign  unique  scores  to  each  bidder. 
However,  it  was  noted  that  the  level  of  commitment  of  a 


successful  bidder  could  change  over  time  and  that  this  issue 
would  have  to  be  worked  at  under  the  contract.  This  sub- 
factor was  probably  the  most  difficult  of  all  to  assess. 

V.3.9  Basis  for  Award 

This  category  discusses  the  outcome  of  the  process 
of  moving  from  the  results  of  the  deliberations  of 
the  evaluation  panel  to  the  selection  of  specific  bid- 
ders. ' It  covers  such  issues  as  whether  the  level  of 
aggregation  of  ratings  enabled  differentiation  among 
the  bidders  on  technical  bases  and  whether  both  the 
technical  and  cost  issues  were  considered  in  making 
the  final  award  decisions. 


The  four  issues  reported  in  this  category  are: 

1 . Did  PFE  provide  for  overall  technical  dis- 
crimination among  bidders? 

Was  there  a sufficient  distribution 
in  the  final  scores  to  determine  a com- 
petitive range? 

2.  Did  PFE  provide  a basis  for  discriminating 
among  bidders  on  specific  technical  issues? 

Did  the  specific  issues  get  washed  out 
by  the  overall  technical  score? 

3.  Was  technical  leveling  a problem? 

Was  technical  transfusion  a problem? 

4.  How  much  of  a factor  was  bid  price  in  the 
final  decision? 

Did  the  procurement  turn  out  to  be  a 
price  competition? 

Did  the  process  provide  a capability 
for  dealing  with  potential  buy-ins? 


V.3.9.1  Overall  technical  discrimination.  One  issue 


of  aggregation  is  whether  PFE  provided  for  overall  technical 
discrimination  among  bidders.  That  is,  whether  there  was 
a sufficient  distribution  in  the  final  scores  to  determine 
a competitive  range.  The  literature  on  source  selection 
indicates  the  overall  scores  are  frequently  close  (e.g., 

Fox,  1974,  p.  269). 

The  distribution  in  the  final  scores  was  fairly  wide, 
though  there  were  no  bidders  at  the  extremes  of  the  scale. 
Figure  V.63  illustrates  the  distributions  which  resulted 
after  multiplying  the  factor  scores  by  the  preassigned 
weights.  On  the  Commercial  Impact  contract,  there  is  a 
marked  difference  between  the  overall  score  of  bidder  101 
and  the  other  two  bidders.  Bidder  101  was  the  only  one 
considered  competitive.  The  results  of  agency  impact  con- 
tract were  less  strikingly  demarcated  but  still  provided 
a strong  basis  for  overall  discrimination.  Bidders  205, 

202,  and  203  were  all  considered  to  be  in  the  competitive 
range,  with  65  points  separating  bidder  205  from  the  other 
two.  Bidder  575  was  120  points  below  202  and  203  and  was 
considered  unacceptable  overall,  as  were  201  and  204  which 
were  further  down  the  scale. 

V.3.9.2  Technical  discrimination  on  specific  issues. 
The  other  issue  of  aggregation  is  whether  PFE  provided  a 
basis  for  discriminating  among  the  competitive  bidders  on 
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FIGURE  V.63  DISTRIBUTION  OF  FINAL  OVERALL  SCORES 


specific  technical  issues,  or  whether  these  got  washed  out 
in  the  overall  score.  This  is  identified  as  a problem  in 
the  literature  (e.g.,  Blue  Ribbon  Defense  Panel,  1970). 

It  is  important  for  clarifications,  negotiation  and  final 
selection  that  the  relative  strengths  of  the  various  bidders 
are  evident.  It  would  be  undesirable  to  select  a bidder 
who  had  a high  overall  score  due  to  great  strengths  in  some 
areas  yet  had  critical  weaknesses  in  other  areas. 

PFE  was  very  strong  at  dealing  with  this  issue.  The 
individual  sub-factor  scores  made  it  easy  to  prepare  for 
clarifications  with  the  competitive  bidders.  The  notes 
in  the  record  indicated  the  specific  questionable  aspects 
of  the  sub-factors  which  the  panel  felt  clarifications  were 
needed  on.  ETIP  was  able  to  probe  the  bidders  for  more 
information  on  these  specific  issues  and  develop  a more 
confident  evaluation  of  the  bidders'  ratings.  A bidder 
with  an  unresolved  extremely  low  score  on  one  sub-factor 
could  have  been  removed  from  the  competitive  range  even 
though  his  overall  score  was  high.  However,  this  case  did 
not  occur. 

Two  of  the  panel  members  felt  that  the  scoring  on  the 
specific  sub-factors  was  the  weakest  aspect  of  the  process 
(see  Figure  V.64).  One  person  felt  that  the  factor/sub- 
factor rating  scale  did  not  provide  for  enough  differentia- 


Responses  From  Panel  Question  6 on  Weakest  Aspect  of  the 
Process 

o The  overall  scoring  aspect  (the  scale).  I don’t  think 
the  sheets  allowed  for  enough  division  if  the  proposals 
were  close  ( PM? ) . 

o It  went  on  too  long  and  there  were  too  many  numbers 
when  they  were  really  unnecessary  (PM4). 

FIGURE  V.64  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  TECHNICAL  DISCRIMINATION  OF  SPECIFIC  ISSUES 

tion  among  bidders  who  were  close.  The  other  felt  that 
there  were  too  many  ratings  (i.e.,  sub-factors)  involved 
in  the  process. 

V.3.9.3  Technical  leveling  and  technical  transfusion. 

An  important  issue  for  the  clarification  and  negotiations 
is  how  to  handle  the  requirement  to  allow  bidders  to  improve 
their  proposals  without  creating  problems  of  technical  trans- 
fusion and  technical  leveling.  Technical  transfusion  arises 
when,  during  clarifications  or  negotiation,  an  approach 
proposed  by  one  bidder  is  suggested  to  another  as  a method 
for  dealing  with  their  technical  problem.  Though  this  is 
illegal  the  literature  indicates  that  it  occurs  frequently 
(e.g.,  Babione,  1978).  If  used  across  all  competitive  bid- 
ders, it  results  in  technical  leveling.  That  is,  all  bid- 
ders eventually- propose  the  same  technical  design  in  order 
to  give  the  government  what  they  feel  it  wants  to  hear. 

PFE  enabled  ETIP  to  deal  with  clarification  and  negoti- 
ation without  problems  of  technical  transfusion.  Since 


clarifications  were  in  terms  of  the  sub-factor  issues  and 
not  in  terms  of  a system  design,  there  was  no  opportunity 
for  transfusion  to  occur. 

During  a discussion  with  the  ETIP's  contract  negotiator 
the  author  raised  the  issue  of  technical  transfusion.  The 
contract  negotiator  indicated  that  it  something  which  a 
contracting  person  always  watches  for  but  that  he  did  not 
remember  it  being  a concern  in  this  source  selection  (Con- 
tract negotiator,  Note  4). 

V.3.9.4  Weight  given  to  bid  price.  An  issue  which 
relates  closely  to  technical  leveling  is  whether  the  source 
selection  turns  into  a price  competition.  If  there  is  no 
basis  for  discriminating  among  competitive  bidders  on  tech- 
nical acceptability,  the  sole  remaining  criteria  is  price. 
That  means  the  lowest  competitive  bidder  always  wins.  This 
encourages  the  practice  of  "buying-in."  "Buying-in"  occurs 
when  a bidder  submits  a proposal  which  is  competitive  at 
the  lowest  cost  and  requests  modifications  once  under  con- 
tract when  it  becomes  clear  to  the  government  that  the  per- 
formance characteristics  or  schedule  cannot  be  met  with 
the  available  resources.  If  all  that  is  available  for  the 
technical  evaluation  is  a final  score,  and  the  final  scores 
are  close,  the  contracting  office  is  at  risk  by  awarding 
to  anyone  other  than  the  low  bidder. 


This  issue  received  a strong  test  in  the  ETIP  case 
and  was  dealt  with  quite  successfully.  On  the  agency  impact 
contract,  there  were  two  bidders  who  were  called  in  for 
clarifications  (see  section  IV.11).  After  oral  and  written 
clarifications,  it  was  quite  easy  to  differentiate  the  rel- 
ative strengths  and  weaknesses  of  these  bidders  on  the  vari- 
ous sub-factors  and  overall.  Had  it  not  been  able  to  obtain 
clarifications  on  the  issues  which  the  panel  was  concerned 
with,  the  contract  would  probably  have  been  awarded  to  the 
other  bidder,  since  they  came  in  with  a lower  cost  estimate. 

V.3.10  Traceability  of  the  Process 

This  category  is  concerned  with  whether  the  evaluation 
can  be  reconstructed  from  the  records.  As  part  of  this, 
it  is  also  concerned  with  whether  the  bidders  perceive  that 
the  process  could  be  easily  traced  and  therefore  difficult 
to  fix.  This  category  is  quite  closely  related  to  the  is- 
sues in  section  V.3.II  on  relationship  with  the  contracting 
office. 

The  four  issues  covered  here  are; 

1 . Did  the  process  communicate  that  there 
was  no  sole  source  position? 

Can  the  process  be  fixed? 

Did  bidders  understand  how  the  decisions 
on  who  wins  actually  get  made? 

2.  Can  changes  in  scores  be  traced  between 
the  initial  ratings  of  panel  members  and 
final  panel  ratings  and  between  final  panel 
ratings  and  the  award  decision? 
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Did  the  panel  follow  the  written  procedure? 


4.  Did  leaving  the  relative  weightings  of 
the  sub-factors  up  to  the  teams  lead  to 
any  problems  (e.g.,  appearance  of  inequi- 
ties)? 

V . 3 . 1 0 . 1 Bidder's  understanding  of  the  evaluation 
process . The  issue  of  whether  the  RF?  communicated  that 
the  process  could  not  be  successfully  fixed  and  that  there 
was  no  sole  source  position  was  important  to  ETIP.  It  was 
thought  to  be  critical  to  obtaining  bids  (section  17.7). 
Since  ETIP  did  not  have  an  established  record  with  evalua- 
tion contractors  and  was  a fairly  small  program. , it  was 
thought  that  there  might  be  problems  attracting  bids  on 
these  projects  with  their  associated  complexity  and  uncer- 
tainty . 

ETIP  generally  did  not  communicate  that  PFE  provided 
any  more  or  less  integrity  to  the  evaluation  than  any  other 
process.  Three  items  were  included  on  the  bidder  question- 
naire to  obtain  perceptions  about  both  understanding  of 
the  evaluation  process  and  whether  it  was  capable  of  corrup 
t ion . 


Did  the  information  provided  enable  you  to  understand  the 
process  which  ETIP  would  use  to  assign  scores  on  each  sub- 
factor? 


Yes  - 5 


No  - 1 


Remark  by  a person  answering  Yes 


o The  amount  of  study  and  cross  referencing  was  very  costly 
for  a small  firm  (BR3). 

Remark  by  person  answering  Mo 

0 There  was  not  sufficient  detail  (3R1). 

FIGURE  V.65  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
5a  ON  BIDDERS’  UNDERSTANDING  OF  THE  EVALUATION 
PROCESS 


Did  the  information  provided  enable  you  to  understand  how 
the  ratings  on  the  various  sub-factors  would  be  combined 
to  arrive  at  final  scores? 

Yes  - 5 No  - I 

FIGURE  V.66  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
5b  ON  BIDDERS’  UNDERSTANDING  OF  THE  EVALUATION 
PROCESS 


Did  you  feel  that  the  process  would  be  a)  difficult 

or  b)  easy  to  corrupt?  i.e.,  For  one  person,  or  a group 

of  people,  who  favored  a specific  bidder  to  exercise  suf- 
ficient control  over  the  process  to  sway  the  award  decision. 

Easy  - 3 Difficult  - 2 Other  - 1 

Remarks  by  those  answering  Easy 

0 There  are  some  people  who  are  domineering  and  since  the 
subjective  judgments  are  altered  in  discussion,  it’s  very 
simple  to  corrupt,  but  no  more  so  than  anything  else  (BR4). 

o By  who  you  select  for  you  panel.  A sophisticated  proposal 
team  is  going  to  try  to  meet  each  of  the  factors.  If 
your  teams  have  had  good  experience  with  a company,  that 
enters  in.  I think  there  should  be  more  emphasis  on  the 
selection  of  the  panel  than  on  the  structure  of  the  pro- 
cess. No  matter  what  quantitative  system  you  have,  'it 
can  be  corrupted  by  the  people  who  are  in  there  (BR2). 

o Any  subjective  review  is  corruptible.  That  did  not  bother 
us.  Maybe  corrupt  is  too  strong  a word.  Influence  may 
be  better  (BR 1 ) . 
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Remarks  by  those  answering  Difficult 

o I really  wanted  a choice  of  c)  no  different  than  any  oth- 
ers. People  who  are  determined  to  corrupt  are  going  to 
corrupt  anything.  If  someone  has  made  up  his  mind  about 
a particular  company,  there  is  always  a way  to  corrupt 
it  (BR4). 

o It  would  be  difficult  unless  the  entire  review  team  fa- 
vored a specific  bidder  (BR6). 

Other 


o It  worked  out  that  way.  We  were  low  bidder  and  had  a 
solid  proposal,  but  the  award  was  made  to  a non-profit. 

I believe  the  award  was  obviously  slanted.  I don't  know 
whether  they  didn't  want  this  company.  I believe  person- 
ally that  they  wanted  an  academic  non-profit  (BR3). 

FIGURE  V.67  BIDDER  REPRESENTATIVES'  RESPONSES  TO  QUESTION 

7 ON  ABILITY  OF  THE  PANEL  TO  CORRUPT  THE  PROCESS 


The  bidders  generally  felt  that: 

o the  information  provided  by  ETIP  enabled  them, 
to  understand  the  process  for  assigning  scores 
to  each  sub-factor  (Yes  - 5 ; No  - 1 ) ; 

o the  information  provided  enabled  them  to  under- 
stand how  the  sub-factor  ratings  would  be  com- 
bined into  a final  score  (Yes  - 5;  No  - 1); 
and 

o the  process  could  be  corrupted  (Easy  - 3;  Dif- 
ficult - 2;  Other  - 1). 

The  remarks  of  four  of  the  six  respondents  to  question 
7 indicate  that  they  believe  the  process  was  subject  to 
corruption  to  the  same  degree  as  any  other  evaluation  pro- 
cess (see  Figure  V.67).  One  of  these  people  felt  that  the 
rigor  applied  to  the  evaluation  did  not  match  that  which 
the  bidder  was  required  to  apply  in  responding  (see  Figure 
V.68).  One  of  the  remaining  two  respondents  felt  that  the 


process,  as  applied,  actually  was  corrupt.  He  felt  the 
oral  clarification  session  was  prejudical  (see  Figure  V.68). 
The  final  respondent  felt  the  process  would  be  difficult 
to  corrupt.  V/hether  some  firms  not  bidding  felt  that  there 
was  a sole  source  position  is  not  known.  As  to  whether 
the  process  could  in  fact  be  easily  fixed,  the  reader  is 
referred  to  the  next  issue  (V. 3. 10.2)  and  the  discussion 
on  how  the  process  actually  worked  in  sections  IV. 10,  V.3.6 
and  V . 3 . 7 . 

V . 3 . 1 0 . 2 Ability  to  trace  scores  through  the  process . 
This  issue  addresses  whether  changes  can  be  traced  from 
the  initial  individual  scores  through  to  the  award  decision. 
This  was  thought  to  be  important  for  keeping  the  process 
honest  and  for  conveying  to  bidders  that  it  was  honest. 

A panel  member  would  probably  be  less  likely  to  try  to  influ- 
ence others  improperly  if  a record  is  kept  of  whether  this 
happened. 

The  records  which  are  kept  by  the  PFE  process  make 
it  quite  easy  to  trace  the  changes  in  scores  between  initial 
individual  ratings  and  the  final  panel  ratings.  The  changes 
in  scores  between  the  final  panel  ratings  and  the  award 
decision  are  traceable  but  less  easily  so. 

The  records  of  the  panel  process  include  the  signed 
rating  sheets  submitted  by  each  panel  member  for  each  sub- 
factor which  they  rated,  tapes  of  the  discussions  by  the 


Response  From  Bidder  Question  1 1 on  the  Weakest  Aspect  of 
the  Process 


o The  evaluation  method.  There  was  sufficient  strength, 
detail,  and  specificity  concerning  the  content  that  a 
contractor  was  to  put  down.  That  the  method  of  evalua- 
tion, as  we  understood  it,  didn’t  seem  to  be  as  rigorous. 
Some  contractors  may  not  have  bid  because  they  felt  the 
COTR  would  make  the  choice  in  the  end  (BR1). 

Response  From  Bidder  Question  13  on  Any  Other  Aspects  of 
the  Process  Which  You  Wish  to  Get  Out 


o The  oral  session  [oral  clarifications]  was  highly  unstruc- 
tured and  prejudicial.  I felt  they  were  rigged  against 
us  (BR3). 


FIGURE  V.68  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  THE  ABILITY  OF  THE  PANEL  TO  CORRUPT  THE 
PROCESS 
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factor  teams,  and  the  signed  final  factor  rating  form. 

To  reconstruct  the  process,  one  only  needs  to  listen  to 
the  tapes  while  having  copies  of  the  rating  forms.  The 
tapes  contain  a record  of  both  the  biases  of  the  panel  mem- 
bers and  the  team  discussions  which  explored  these  biases 
in  orde^r  to  work  toward  agreement  upon  final  scores. 

The  documentation  which  is  available  for  the  later 
portions  of  the  process  includes  the  memorandums  submitted 
to  the  contract  negotiator,  the  tapes  of  the  oral  clarifica- 
tion sessions,  and  the  written  responses  of  the  bidders 
to  the  questions  for  clarifications.  From  the  memos  and 
tapes  it  is  possible  to  trace  how  the  issues  which  the  panel 
raised  were  followed  up  in  the  clarification  process.  The 
one  part  of  the  process  on  which  there  is  not  a strong  record 
is  the  deliberations  involved  in  re-evaluating  sub-factor 
ratings  based  on  the  new  information  obtained  during  clar- 
ifications. The  actual  new  ratings  are  available  but  there 
is  not  a taped  record  of  the  discussions  as  there  is  with 
earlier  parts  of  the  process.  In  order  to  recreate  this 
part  of  the  process,  it  would  be  necessary  to  find  the 
changes  between  the  final  sub-factor  scores  and  the  scores 
assigned  by  the  panels  and  compare  these  with  the  new  infor- 
mation which  was  made  available  during  the  clarifications. 

V. 3. 10.3  Degree  to  which  the  rules  were  followed. 

An  issue  related  to  traceability  is  whether  the  panel  mem- 
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bers  actually  followed  the  evaluation  procedure  which  the 
RFP  said  would  be  followed  or  whether  they  changed  the  pro- 
cess or  criteria  in  mid-evaluation.  There  are  two  ways 
in  which  the  panel  might  alter  the  rules.  One  is  that  dur- 
ing the  evaluation  process  the  panel  might  find  that  one 
or  more  of  the  bidders  have  raised  issues  and  proposed  ap- 
proaches which  the  * government  did  not  include  in  the  RFP. 
The  panel  might  then  proceed  to  include  these  issues  in 
their  evaluation  of  all  of  the  other  proposals.  The  other 
problem  arises  if  the  panel  finds  that  it  cannot  work  with 
the  evaluation  criteria  which  were  included  in  the  RFP. 

They  may  be  either  too  specific  or  not  specific  enough. 

The  panel  might  then  make  up  new  criteria  as  it  goes  along. 

ETIP  had  no  problems  with  this  issue.  The  sub-factors 
provided  the  issues  for  the  bidders  to  respond  to  and  were 
the  units  where  measurements  of  acceptability  were  taken. 
The  criteria  of  understanding,  approach,  and  capability 
were  used  throughout  the  evaluation.  Had  the  panel  discov- 
ered that  a sub-factor  was  missing  it  could  have  been  sent 
to  all  bidders  as  an  amendment  to  the  RFP.  However,  this 
did  not  arise. 

One  point  which  should  be  noted  is  that  the  sub-factor 
rating  process  as  described  in  RFP  sounds  more  sequential 
than  it  worked  in  practice.  The  RFP  stated: 

Members  of  the  proposal  evaluation  team  concerned 

with  each  factor  will  meet  and  perform  a joint 
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evaluation.  They  will  follow  an  established  pro- 
cedure entitled,  "Factor  Team  Guidelines,"  to 
assure  an  orderly  and  efficient  process.  Briefly, 
they  will  first  compare  notes  informally  to  assure 
common  standards  and  information;  second,  they 
will  arrange  the  bidders  in  descending  order; 
third,  they  will  establish  the  relative  spread 
among  the  bidders;  and  finally,  their  result  will 
be  permanently  recorded  and  signed  by  all  team 
members  present...  (Thompson,  1976). 

As  can  be  seen  from  the  anectodes  in  section  IV. 10,  all 
of  these  "steps"  occurred  during  more  or  less  in  the  same 
time  frame  and  were  also  somewhat  iterative.  The  team  meet- 
ings would  begin  with  a review  of  the  evaluation  procedure, 
for  the  benefit  of  new  team  members.  Next  the  individual 
scores  for  the  first  set  of  sub-factors  would  be  posted. 

In  general,  the  panel  would  then  discuss  the  bidders,  one 
at  a time,  and  work  to  find  an  appropriate  score.  The  order 
* and  spread  would  both  be  discussed  at  the  same  time.  In 
some  cases  the  panel  would  discuss  whether  they  should  re- 
vise the  ranking  of  a bidder  covered  early  in  the  discus- 
sion, to  reflect  the  panel’s  improved  level  of  certainty 
with  respect  to  the  spread. 

Another  illustration  of  the  panel  holding  to  the  pro- 
cedure was  given  in  section  IV. 10.8.  At  one  of  the  later 
factor  team  meetings  of  the  agency  impact  evaluation  system 
contract,  a relatively  senior  panel  member  (from  outside 
of  ETIP)  suggested  that  time  could  be  saved  by'  determining 
which  bidders  were  already  outside  of  the  competitive  range 
and  excluding  them  from  further  discussion.  Another  panel 


member  pointed  out  that  the  rules  of  the  procedure  did  not 
allow  for  this.  The  panel  continued  rating  all  of  the  bid- 
ders. This  has  some  significance  for  evaluating  this  issue 
since  it  points  to  how  people  who  are  experienced  in  propo- 
sal evaluation  may  quite  casually  suggest  alterations  in 
the  procedures  once  the  evaluation  is  under  way. 

V. 3. 10.4  Weighting  of  the  sub-factors.  The  last  issue 
covered  under  this  category  concerns  whether  leaving  the 
relative  weightings  of  the  sub-factors  up  to  the  teams  led 
to  any  inequities  or  appearances  of  inequities.  This  was 
a major  issue  raised  by  some  ETIP  and  PAA  staff  prior  to 
the  evaluation  (see  section  IV. 8).  It  was  feared  that  leav- 
ing the  decision  on  the  relative  weightings  up  to  the  indi- 
vidual teams  would  present  the  ability  to  improperly  bias 
the  evaluation  for  or  against  specific  bidders. 

The  process  of  weighting  sub-factors  did  not  seem  to 
create  any  inequities.  This  is  discussed  in  V.3.6.5,  under 
the  issue  on  time  required  to  agree  on  sub-factor  weights. 
Also,  the  process  of  weighting  generally  did  not  present 
the  appearance  of  inequities.  An  item  was  included  in  the 
bidder  questionnaire  which  asked  whether  the  respondent 
was  concerned  that  not  specifying  relative  weightings  of 
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the  sub-factors  might  lead  to  inequities.  The  response 
was:  Yes  - 1 ; No  - 5 (see  Figure  V.69).  The  one  person 
answering  yes  thought  that  the  entire  process  would  lead 


to  inequities  since  there  were  no  predetermined  criteria 
for  each  scale  score. 


Were  you  concerned  that  not  specifying  the  relative  v/eight- 

ings  of  the  sub-factors  might  lead  to  inequities  in  the 

process? 

Yes  - 1 Mo  - 5 

Remark  by  person  answering  Yes 

0 The  whole  process  would  lead  to  inequities.  You’re 
taking  nonparametr ic  data  and  trying  to  make  it  para- 
metric. There  were  no  boundary  conditions  for  scoring 
so  that  someone  could  say  that  an  A contains  the  fol- 
lowing five  items  or,  here  is  an  example  of  an  A vs. 
an  F (BR5). 

Remarks  by  those  answering  No 

o Inequities  was  not  the  issue.  We  were  concerned  about 
accuracy  (BR4). 

0 The  teams  probably  had  too  much  instruction  as  it  was. 

I didn’t  see  that  as  a problem  (BR2). 

FIGURE  V.69  BIDDER  REPRESENTATIVES’  RESPONSES  TO  QUESTION 
6 ON  WEIGHTING  OF  SUB-FACTORS 


V.3.11  Relationship  with  the  Contracting  Office 
A category  of  issues  which  is  perhaps  most  important 
to  achieving  timely  award  of  contracts  is  the  relationship 
between  the  technical  program  staff  and  the  contracting 
office.  Since  the  contracting  officer  is  the  one  whose 
name  goes  on  the  contract  and  is  legally  responsible,  it 
is  in  the  interest  of  contracting  office  personnel  to  pro- 
ceed cautiously  and  not  allow  themselves  to  be  rushed  by 
program  people  who  are  anxious  to  meet  their  deadlines. 


In  ETIP’s  case,  the  use  of  PFE  was  a departure  from  more 
conventional  source  selection  techniques.  As  such,  it  had 
the  potential  of  leaving  the  contracting  office  open  to 
protests  from  losing  bidders.  People  at  ETIP  were  concerned 
that  the  process,  with  its  flexible  statement  of  work,  was 
not  legal  or  that  the  contracting  officer  would  not  allow 
ETIP  to  use  it  for  fear  of  protests. 

The  issues  covered  under  this  category  are: 

1 . Is  PFE  legal  under  the  Federal  Procurement 
Regulations? 

2.  Did  the  contracting  officer  feel  that  the 
selections  mads  b'y  using  the  process  would 
be  defensible  if  there  were  a GAO  investi- 
gat ion? 

3.  Did  the  contract  negotiator  accept  the  evalua- 
tion of  the  panel? 

Did  he  want  the  panel  to  revise  their  re- 
port? » 

Did  he  want  another  level  of  technical 
review  of  the  panel’s  evaluation? 

4.  a)  Did  using  PFE  result  in  timely  award? 

b)  Were  there  any  protests  of  the  award? 

c)  Were  any  debriefings  requested? 

V. 3. 11.1  PFE  and  the  Federal  Procurement  Regulations. 

A key  issue  to  resolve  was  whether  PFE  was  legal  under  the 
Federal  Procurement  Regulations  (FPR).  This  was  established 
early  in  a January  27  meeting  at  the  contracting  office. 

The  contracting  officer  had  a background  in  systems  procure- 
ment and  recognized  immediately  that  there  would  be  no  legal 
problems . 
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The  FPR  allows  for  a great  deal  of  flexibility  in  the 
source  selection  process  as  long  as  its  minimal  requirements 
are  met.  PFE  did  not  go  against  the  grain  of  the  FPR  (Con- 
tract negotiator,  Note  5). 

V.3.11.2  Ability  to  defend  under  investigation.  The 
next  issue  was  whether  the  contracting  officer  felt  that 
the  selections  made  by  using  the  PFE  process  would  both 
appear  honest  and  be  defensible  under  a GAO  investigation. 

It  was  important  that  the  RFP  convey  sufficient  detail  about 
the  evaluation  process  and  the  documentation  produced  that 
bidders  would  realize  that  it  was  not  only  a fair  process 
but  that  ETIP  would  have  no  difficulty  proving  this  under 
a protest  investigation. 

During  a discussion  about  this  source  selection  with 
the  contract  negotiator  he  indicated  that  the  major  concern 
of  contracting  people  is  whether  there  are  any  protests. 

To  have  a protest  means  that  you  have  done  something  so 
that  people  feel  that  they  have  an  opening  to  justify  a 
protest.  One  measure  of  how  well  you’ve  done  is  whether 
you  get  a protest  (Contract  negotiator,  Note  6).  The  con- 
tract negotiator  indicated  that  he  was  very  pleased  about 
the  results  of  this  source  selection  since  there  were  no 
protests.  This  was  especially  true  since  one  bidder,  who 
was  thoroughly  versed  in  procurement  law  and  practice,  could 
not  find  a basis  for  protest,  even  though  he  seemed  very 
angry  about  losing  (Contract  negotiator.  Note  7). 
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In  discussion,  the  contract  negotiator  indicated  that 
he  was  initially  skeptical  about  whether  or  not  the  process 
would  produce  protests.  It  was  novel  and  of  greater  magni- 
tude than  other  processes  (Contract  negotiator,  Note  8). 

A key  reason  he  was  willing  to  use  the  method  was  because 
he  had  faith  in  the  technical  people  at  ETIP  whom  he  had 


previously  worked  with.  He  knew  that  they  had  a sensitivity 
to  the  areas  where  the  contracting  office  might  be  vulner- 
able (Contract  negotiator.  Note  9).  Also,  he  realized  that 
the  RFP  let  the  company  know  what  they  had  to  respond  to 
and  that  the  bidder  would  know  that  the  inform.ation  would 


reach  the  evaluators.  If  there  were  an  oversight  it  would 
be  the  bidder's  and  not  ETIP's  (Contract  negotiator.  Mote 


10)  . 


Another  thing  which  helped  to  establish  his  confidence 

in  the  process  was  the  response  from  firms  which  received 

the  RFP.  He  indicated  that: 

A contracting  officer  can't  anticipate  all  of 
the  problems,  especially  when  they  are  of  a techni- 
cal nature.  So,  you  rely  a lot  on  the  private 
sector  to  come  back  and  ask  questions  and  raise 
concerns.  You  get  a feeling  at  that  point  in 
time  on  how  well  you've  done  your  hom.ework  and 
how  well  the  technical  people  you've  relied  on 
have  done  their  homework  (Contract  negotiator. 

Note  11). 

0 

Also,  he  felt  that  it  was  clear  that  it  would  be  difficult 


for  one  person  to  influence 
well  expressed  in  the  RFP. 
collusion  for  that  to  work 


the  panel  and  that  this  was 
Everyone  would  have  to  be  in 
Contract  negotiator.  Note  12). 


The  contract  negotiator's  final  skepticism  about  wheth- 
er the  process  would  work  disappeared  after  the  panel  report 
came  in.  At  that  point  it  was  clear  to  him  that  it  would 
stand  up  under  any  investigation  if  it  went  to  the  GAO  (Con- 
tract negotiator,  Mote  13).  He  felt  that  the  strongest 
aspect  of  the  process  was  that,  from  a technical  evaluation 
standpoint,  "when  you  get  down  to  the  bottom  line,  there 
is  no  doubt  that  it  is  going  to  stick"  (Contract  negotiator. 
Mote  14). 

V.3.11.3  Revisions  of  panel  results.  Another  issue 
which  came  up  prior  to  the  proposals  being  received  was 
the  MBS  administration's  suggestion  that  a committee  be 
set  up  to  review  the  findings  of  the  panel.  This  might 
have  negated  some  of  the  work  of  the  panel. 

ETIP  had  no  problems  in  having  the  report  of  the  pan- 
el's findings  accepted.  When  the  COTR  told  the  contract 
negotiator  of  the  MBS  request  for  a review  panel,  the  con- 
tract negotiator  indicated  that  he  did  not  wish  to  have 
this  intermediate  step  put  into  the  process.  When  the  COTR 
submitted  his  memorandum  to  the  contract  negotiator,  there 
were  no  requests  for  revisions. 

During  subsequent  discussions,  the  contract  negotiator 
indicated  his  reasons  for  not  wanting  a review  panel.  He 
said  that  he  felt  that  the  evaluation  process  which  was 
established  was  clear  and  thorough.  To  have  another  level 


of  review,  which  could  negate  the  panel's  recommendations, 
would  open  the  process  to  criticism  and  probably  couldn't 
be  supported  (Contract  negotiator,  Note  15). 

He  felt  that  the  report  from  the  COTR  was  very  thorough 
and  documented  with  respect  to  specific  points.  He  added 
that  bidders  normally  aren't  told  what  they  are  going  to 
be  evaluated  on  beyond  the  specific  factors.  This  provided 
the  capability  to  get  down  to  sub-factors  (Contract  negoti- 
ator , Note  1 6 ) . 

In  elaborating  on  his  reaction  to  the  COTR ' s report, 

the  contract  negotiator  said: 

Sometimes  [programs]  don't  support  their  recommen- 
dations sufficiently.  You  have  to  ask  them  why 
they  want  to  negotiate  with  these  two  but  not 
this  one  who  is  only  five  points  lower.  They 
don't  have  specific  reasons  documented.  'Vhen 
you  came  back  and  said  that  these  guy^  are  accept- 
able or  unacceptable  on  this  sub-factor,  I believed 
it.  A lot  of  times  when  you're  told  that  a guy 
is  technically  acceptable  you  are  not  sure  they 
can  support  it  or  whether  they  have  [just  worked 
up  the  score]  (Contract  negotiator.  Note  17). 

V.3.11.4  Timeliness  of  award.  The  final  issue  in 

this  category  concerns  the  timeliness  of  the  award  and  wheth' 

er  there  were  any  protests  which  delayed  the  start  of  the 

work . 

ETIP  was  successful  in  meeting  its  deadline  of  the 
end  of  the  fiscal  year  and  there  were  no  protests.  There 
was  a request  for  one  debriefing.  The  bidder  making  the 


request  was  clearly  upset  at  having  lost,  since  he  consi- 


dered  himself  most  qualified.  He  let  it  be  known  that  he 
felt  that  ETIP  made  the  wrong  choice.  However,  he  did  not 
choose  to  file  a formal  protest.  The  contract  negotiator 
felt  good  about  the  debriefing  process  since  the  COTR  was 
able  to  tell  the  bidder  what  his  strong  and  weak  points 
were  (Contract  negotiator,  Note  18). 

V.3.12  Resource  Requirements 

The  category  of  resource  requirements  deals  with  issues 
of  the  costs  of  the  source  selection  to  the  government  and 
the  bidders.  These  costs  are  considered  in  terms  of  the 
total  amounts  of  time  required  by  various  personnel,  the 
dollar  costs  of  responding  to  the  RFP,  and  the  length  of 
time  over  which  these  resources  were  spent. 

There  are  four  issues  covered  in  this  section.  They 

are : 

1 . How  much  time  was  required  to  do  system  or 
evaluation  design  work  prior  to  issuing  the 
RFP? 

o elapsed 
o total 

2.  How  much  time  did  it  take  for  bidders  to 
respond  to  the  RFP? 

3.  How  much  time  did  it  take  for  the  panel  to 
evaluate  proposals? 

How  much  time  did  it  take  to  read  propo- 
sals? 

How  much  time  did  the  panel  meetings  (and 
preliminary  meetings)  take? 

How  much  time  would  oral  presentations 
have  taken? 


4. a)  How  long  did  it  take  to  make  an  award  once 
the  panel's  evaluations  were  complete? 

How  much  time  did  it  take  to  write  up  the 
results  of  the  panel  meetings? 

How  long  did  it  take  to  prepare  for  oral 
clarif ications? 

How  long  did  it  take  to  request  written  con- 
firmations or  oral  clarifications? 

How  long  did  it  take  to  reach  a decision 
once  the  written  clarifications  were  re- 
ceived? 

b)  How  long  did  it  take  to  prepare  for  the  de- 
briefing? 

V. 3.12.1  Resource  required  prior  to  issuing  the  RFP . 
The  first  issue  under  this  category  concerns  the  resources 
required  by  ETIP  to  write  and  issue  the  RFP.  The  total 
elapsed  time  between  the  time  when  the  COTR  asked  Thompson 
to  write  the  RFP  until  the  time  when  the  draft  RFP  was  ready 
for  distribution  was  four  months.  Most  of  the  actual  time 
required  in  writing  the  RFP  was  Thompson's,  though  he  held 
some  discussions  with  other  ETIP  staff  to  help  him  define 
the  issues.  The  time  to  write  the  RFP  probably  shortened 
somewhat  since  Thompson  used  the  Wright  Field  RFP  as  a base 
for  the  management  sub-factors,  the  Proposal  Preparation- 
Procedure,  Introductory  Notes,  and  the  Scope  of  Effort  Docu- 
ments. Had  the  process  of  getting  program  participants 
involved  in  formulating  the  sub-factors  been  used,  the  a- 
raount  of  time  contributed  by  others,  and  probably  the  over- 
all elapsed  time,  would  have  been  greater. 

V. 3. 12.2  Resources  required  for  bidders  to  respond. 

The  issue  of  the  cost  to  bidders  of  responding  is  important 

First,  some  qualified  bidders  may  decide 


in  two  respects. 


that  the  expected  value  of  bidding  (cost  of  bidding  weighed 
against  subjective  probability  of  winning  and  value  of  win- 
ning) is  not  great  enough,  causing  the  government  to  not 
get  the  bids  it  desires.  Second,  much  of  the  cost  of  re- 
sponding is  eventually  paid  for  by  the  government  through 
reimbursable  overhead  on  other  contracts.  Though  this  does 
not  show  up  as  a direct  cost  on  the  source  selection,  it 
does  increase  the  costs  of  government  procurements.  This 
problem  is  related  to  the  issue  of  having  bidders  be  able 
to  self-select  effectively  on  whether  or  not  to  bid  (section 
V.3.3.2)  so  that  large  numbers  of  unqualified  firms  do  not 
invest  in  responding. 

Some  people  felt  that  the  process  was  quite  costly 
for  bidders  to  respond  to.  The  author  has  heard  estimates 
of  cost  to  respond  ranging  from  $10,000  to  $20,000.  This 
can  be  quite  a burden  for  a small  firm  (see  Figure  V.11 
and  V.70).  However,  with  clear  criteria  for  the  evaluation, 
the  firms  should  be  better  able  to  assess  the  risk. 


Response  From  Bidder  Question  11  on  the  Weakest  Aspect  of 
the  Process 

o For  small  profit  making  firms  it  was  just  too  much 
of  a good  thing.  The  same  thing  could  have  been  done 
without  all  of  the  tasks  and  sub-tasks.  It  could  have 
been  made  simpler  and  less  costly  for  small  firms. 

FIGURE  V.70  RESPONSES  TO  OPEN  ENDED  QUESTIONS  WHICH  RELATE 
TO  RESOURCES  REQUIRED  FOR  BIDDERS  TO  RESPOND 
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The  cost  required  must  be  weighed  against  dollar  value 
of  the  contract  and  the  risk  of  making  a selection  by  using 
a less  comprehensive  (and  less  costly)  procedure.  ETIP 
felt  that  the  risk  justified  using  the  procedures. 

V . 3 . 1 2 . 3 Resources  required  for  panel  to  evaluate 
proposals . A major  portion  of  the  government  resources 
devoted  to  source  selection  is  in  the  evaluation  of  the 
proposals  by  the  panel.  This  includes  the  tim.e  to  learn 
the  evaluation  process,  the  time  to  read  the  proposals  and 
assign  initial  ratings,  and  the  time  for  the  evaluation 
panel  meetings.  One  of  the  major  concerns  of  some  of  the 
ETIP  and  PAA  staff  was  that  the  time  required  for  this  would 
be  on  the  order  of  months  (section  IV . 6 ) . 

This  was  one  of  the  areas  where  there  was  a clear  ad- 
vantage by  using  PFE.  The  tim.e  to  learn  the  evaluation 
process  was  minimal  (see  section  V.3.13).  No  extra  reading 
period  was  required  in  the  schedule.  The  proposals  were 
distributed  on  a Friday  and  the  first  factor  team  was  pre- 
pared to  meet  on  Monday.  The  panel  members  were  able  to 
accomplish  all  of  their  reading  over  the  weekend,  at  night, 
or  during  the  meetings  of  team.s  on  which  they  were  not  mem- 
bers., All  fourteen  factor  team  meetings  (seven,  for  each 
contract)  were  completed  over  a one  week  period.  The  meet- 
ing times  ranged  from  one  hour  to  two  hours  and  35  minutes, 
with  the  average  meeting  lasting  one  and  a half  hours  (see 


Figure  V.31).  In  all,  approximately  100  hours  of  total 

time  of  various  people  was  taken  for  meetings  to  evaluate 

14 

the  proposals  for  both  contracts  ( Yj  where  is  the 

number  of  people  on  the  i^  team  and  1^  is  the  approximate 
length  of  the  i^^  team  meeting).  Specialists  who  partici- 
pated on  only  one  team  were  able  to  limit  their  time  in 
factor  team  meetings  to  one  or  two  hours. 

The  time  of  people  to  read  appropriate  parts  of  the 
proposals  (which  is  not  known)  must  be  added  this  to  actu- 
ally determine  the  total  amount  of  time  required. 

The  amount  of  time  required  by  the  government  to  evalu- 
ate the  proposals  seemed  minimal  given  the  significance 
and  magnitude  of  the  contracts. 

If  the  original  plan  for  having  bidders  make  oral  pre- 
sentations to  the  panels  had  been  followed,  the  total  amount 
of  meeting  time  required  by  the  panel  would  probably  have 
at  least  tripled. 

V. 3. 12.4  Resources  required  following  the  panel*s 
evaluation.  One  of  the  issues  in  source  selection  is  that 
it  frequently  takes  a considerable  amount  of  time  to  reach 
an  award  decision  and  sign  a contract  once  the  panel’s  eval- 
uation is  complete.  This  can  be.  taken  up  reviewing  cost 
proposals,  obtaining  clarifications,  preparing  and  revising 
written  justifications  for  decisions  reached,  and  negotiat- 
ing with  bidders  considered  competitive.  In  some  instances 
this  can  go  on  for  many  months. 


In  ETIP's  case  the  award  was  made  a little  over  two 
months  following  the  completion  of  the  last  panel  meeting. 
The  memorandum  summarizing  the  panel's  ratings  was  sent 
to  the  contracting  office  11  days  after  the  last  meeting 
(May  4,  1976).  The  oral  clarification  sessions  were  held 
three  weeks  later  (May  25-26,  1976).  Questions  for  written 
clarification  were  sent  out  on  June  4 with  responses  due 
on  June  11.  The  COTR's  recommendations  were  sent  to  the 
contracting  office  on  June  17.  The  contracting  office's 
negotiations  took  two  additional  weeks  and  the  contracts 
were  signed. 

A debriefing  was  held  with  one  of  the  unsuccessful 
bidders  a month  after  the  signing  of  the  contracts.  It 
was  given  by  the  contract  negotiator  with  the  assistance 
of  the  COTR  and  the  author.  The  preparation  by  the  COTR 
and  the  author  consisted  of  reviewing  their  notes  and  the 
records  from  the  evaluation  on  the  afternoon  before  the 
meeting . 


V.3.13  Education  of  Government  Personnel 
The  category  of  the  Education  of  Government  Personnel 
is  concerned  with  whether  the  PEE  process  provided  training 
and  education  to  government  personnel  which  would  help  them, 
in  both  the  evaluation  of  proposals  and  in  the  management 
of  the  subsequent  contracts. 


CO 


The  two  issues  covered  here  are: 


1.  Did  use  of  PFE  result  in  education  of  govern- 
ment personnel  so  that  they  would  be  able  to 
effectively  evaluate  the  bidders'  proposals? 

2.  Did  use  of  PFE  result  in  education  of  govern- 
ment personnel  so  that  they  would  be  better 
able  to  work  with  or  manage  the  resulting  con- 
tractors? 

Were  program  participant^  willing  to  be 
involved  in  the  procurements? 

V. 3. 13.1  Education  provided  for  evaluating  proposals. 
The  issue  of  providing  education  in  source  selection  to 
government  personnel  prior  to  proposal  evaluation  is  one 
of  enabling  them  to  have  a sufficient  knowledge  of  the  pro- 
ject, the  RFP,  and  the  evaluation  process  so  that  they  will 
be  able  to  perform  effectively  when  the  proposals  arrive. 
They  must  understand  which  sections  of  the  proposals  to 
read  and  how  to  apply  the  criteria  for  evaluation  to  the 
sections  for  which  they  are  responsible. 

The  panel  seemed  to  be  well  prepared  for  the  evaluation 
given  small  amount  of  prior  preparation  which  occurred. 

Most  of  the  information  concerning  the  project  and  the  eval- 
uation process  was  conveyed  to  panel  members  through  the 
copies  of  the  RFP  documents  which  were  distributed  during 
the  month  before  the  proposals  were  due  and  through  the 
short  briefing  the  day  after  the  proposals  were  received. 
Some  of  the  panel  members  also  attended  a meeting  a month 
before  the  proposals  were  due  during  which  the  evaluation 
process  was  reviewed  (see  section  IV. 8). 
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At  each  factor  team  meeting  where  new  panel  members 
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V. 3.13*2  Education  provided  for  working  with  winning 
bidders . The  issue  of  whether  training,  learning,  and  educa- 
tion was  provided  for  the  governm.ent  personnel  who  would 
work  on  the  contract  was  thought  to  be  critical  to  ensure 
a progressive  and  successful  accomplishment  of  the  objec- 
tives of  the  procurement  (Thompson,  1976a).  It  was  equally 
important  for  the  governm.ent  people  to  understand  the  issues 
involved  in  achieving  the  objectives  as  it  was  for  the  suc- 
cessful bidder. 

The  author  believes  that  ETIP  was  only  minimally  suc- 
cessful in  dealing  with  this  issue.  One  reason  was  that 
only  one  PAA  staff  member  participated  in  the  evaluation 
and  two  key  ETIP  staff  did  not  join  the  program  until  immedi- 
ately after  the  signing  of  the  contracts.  Perhaps  of  even 
greater  significance  was  that  the  government  personnel  who 
would  work  on  the  project  were  not  involved  in  the  process 


of  selecting  and  writing  the  sub-factors.  This  is  a part 
of  the  PFE  process  which  was  foregone  due  to  time  con- 
straints. It  is  likely  that  significantly  more  education 
would  have  occurred  had  this  process  been  followed. 

V.3.14  Education  of  Bidders 

The  category  of  education  of  bidders  is  concerned  with 
whether  the  process  of  responding  to  the  RFP  provided  infor 
mation  to  bidders  which  was  useful  in  their  future,  interac- 
tions with  ETIP.  In  the  case  of  the  successful  bidders 
the  issue  is  whether  education  was  provided  which  would 
enable  them  to  perform  more  effectively  on  the  contract. 

In  the  case  of  the  unsuccessful  bidders,  the  issue  is  wheth 
er  education  was  provided  which  would  help  them  in  respond- 
ing or  deciding  not  to  respond  on  future  ETIP  RFPs.  Only 
the  former  issue  is  covered  here. 

The  issue  of  whether  training,  learning,  and  education 
was  provided  for  the  successful  bidders  during  the  source 
selection  process  was  thought  to  be  critical  to  the  accom- 
plishment of  the  objectives  of  the  procurement  (Thompson, 
1976a).  It  was  important  for  the  successful  bidders  to 
be  aware  of  what  was  known  about  the  procurement,  to  under- 
stand what  the  government  felt  the  important  issues  were, 
and  to  have  some  approaches  for  dealing  with  these  issues 
(including  how  to  improve  their  capabilities  in  areas  where 
they  realized  that  they  were  weak).  It  was  also  important 


for  the  successful  bidders  to  have  a good  understanding 
of  the  Statement  of  Work  and  how  it  would  be  satisfied 
through  the  successive  approvals  and  acceptances  process. 

The  two  successful  bidders  were  asked,  in  the  bidder 
questionnaire,  whether  responding  to  the  RFP  gave  an  early 
understanding  of  how  to  deal  with  the  issues  v/hich  were 
important  to  successful  performance  on  the  contract.  One 
answered  yes  and  the  other  no.  The  bidder  answering  no 
remarked  that: 

We  were  not  prepared  for  the  preoccupation  of 
ETIP  management  with  the  evaluation  of  exper  im.ents , 
where  the  initial  problem  of  design,  initiation 
and  execution  of  experiments  were  not  adequately 
addressed  by  ETIP  management. 

One  of  the  failures  of  this  application  of  PFE  was 
the  understanding  of  the  statement  of  work  and  how  it  would 
be  satisfied.  The  RFP  contained  both  the  brief  statement 
of  work  which  specified  only  that  periodic  and  final  reports 
would  be  delivered  (with  no  schedule)  and  a longer  statement 
of  work  containing  lists  of  illustrative  reports  and  an 
accompanying  proposed  schedule.  From  the  early  performance 
of  the  contractors  and  the  ETIP  staff,  it  was  clear  that 
everyone  was  trying  to  comply  with  the  illustrative  report- 
ing schedule.  This  created  unintended  em.phasis  on  the  re- 
ports as  opposed  to  developing  the  systems. 

V.3.15  Additional  Comments  from  the  Questionnaires 
Some  of  the  remarks  which  were  made  by  respondents 
to  the  open  ended  questions  do  not  fit  properly  under  any 


of  the  specific  issues  covered  in  the  preceeding  sections. 
They  are  of  a more  general  nature.  These  remarks  are  listed 
in  Figure  V . 7 1 . 

Some  of  these  comments  provide  additional  perspective 
for  understanding  answers  given  by  the  respondents  to  the 
other  questions.  For  example,  in  BR5’s  answer  to  the  ques- 
tion about  the  strongest  aspect  of  the  process  he  said  that, 
"I  didn’t  think  there  were  any."  In  response  to  the  ques- 
tion which  asked  whether  there  was  any  other  aspect  of  the 
process  which  he  wished  to  mention  he  said,  "I  think  it 
was  theoretically  unsound  and  without  basis."  The  author 
followed  up  and  asked  why  he  bid  if  he  felt  that  way.  His 
answer  was:  "Because  its  money  and  my  boss  said  to  do  it. 

If  I had  free  choice  I wouldn’t  have  bid." 

V.4  Summary 

This  section  presents  a summary  of  the  results  of  ETIP’ 
use  of  the  Parametric  Factor  Evaluation  process  for  its 
source  selection.  The  section  is  divided  into  a summary 
for  those  issues  for  which  the  evaluation  indicates  that 
ETIP’s  use  of  PFE: 

o proved  successful, 

» 

o proved  unsuccessful,  and 
o are  equivocal. 


Responses  to  Panel  Question  5 on  the  Strongest  Aspect  of 
the  Process 
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o The  fact  that  we  got  a good 
able  to  get  an  experimental 
cracy  ( PMl 5 ) . 


contractor  and  that  we  were 
procurement  through  the  bureau- 


Responses  to  Panel  Question  6 on  the  Weakest  Aspect  of  the 
Process 


o The  panel  was  not  involved  up  front  in  develooine  the 
RFP  (PM11). 

o The  relative  lack  of  early  input  (and  commitment  from 
stakeholders,  e.g.,  PAA ’ s and  MBS)  (PM13). 

Responses  to  Panel  Question  10  on  Any  Other  Aspects  of  the 
Process  Which  You  Wish  to  Get  Out 


o Mot  having  a really  solid  conception  of  what  we  were  loo 
ing  for  and  where  we  were  going  and  why  Phases  II  and  II 
were  important  and  other  things  influenced  the  process. 

It  probably  didn't  affect  the  outcome  in  the  sense  of 
which  offer  we  selected.  It  may  have  caused  us  to  probe 
on  certain  issues  and  get  certain  commitments  up  front 
(PMl 1 ) . 

0 I felt  the  evaluation  procedures  were  theoretically  excel- 
lent. It  was  a little  difficult  operationally  (PMl). 

o There  might  have  been  a little  too  much  detail,  but  I 
didn’t  know  the  issues.  That  was  for  you  to  say  (?M6). 

Responses  to  Bidder  Question  10  on  the  Strongest  Aspect 

of  the  Process 

o I didn’t  think  there  were  any  (BR5). 
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o The  evaluation  procedure  which  ETIP  used  was  the  strongest 
aspect.  No  particular  aspect  (BR6). 

Responses  to  Bidder  Question  11  on  the  Weakest  Aspect  of 
the  Process 

o I didn’t  see  the  basis  for  doing  it.  I think  it  failed 
to  look  at  the  theory  they  were  espousing  at  the  time. 

I fail  to  see  where  the  notions  that  they  espoused  in 
terms  of  Campbell  and  Stanley  were  used  to  design  this 
process  (BR5). 

o I have 'no  problem  with  your  evaluation  process.  The  prob- 
lem is  that  the  ETIP  program  had  flaws  and  there  was  noth- 
ing coming  up  to  evaluate.  The  RFP  did  not  reflect  the 
shortcomings  of  ETIP  management.  The  information  was 
voluminous- and  provided  us  with  a good  array  of  informa- 
tion up  to  that  point.  I don't  think  we  would  have  felt 
we  were  treated  unfairly  if  we  had  lost.  Under  the  work 
you  were  too  preoccupied  with  evaluation  (BR6). 

0 Absence  of  a substantive  and  policy  understanding  of  the 
problem  they  were  approaching  in  relationships  with  mul- 
tiple agencies  and  the  use  of  softer  designs.  There  was 
a greater  emphasis  on  quasi-experimental  designs  where 
the  issue  was,  how  were  you  going  to  do  some  good  case 
studies  and  descriptive  materials  (BR2). 

Responses  to  Bidder  Question  13  on  Any  Other  Aspects  of 
the  Process  Which  You  Wish  to  Get  Out 


0 I think  it  was  theoretically  unsound  and  without  basis 
(BR5). 

o There  was  a kind  of  euphoria  and  a kind  of  pride  in  the 
RFP,  by  the  people  who  prepared  it,  that  I don’t  think 
would  have  been  equally  shared  by  old  crusty  government 
contractors  (BR4). 

o ETIP  didn't  seem  to  be  as  much  in  control  as  Charlie  Thomp- 
son. I think  most  of  the  people  who  were  bidding  got 
the  feeling  that  he  was  handling  his  graduate  students 
and  he  reflected  a lack  of  knowledge  of  government  (BR2).. 
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Chapter  VI  presents  summary  findings  from  the  evaluation 
which  go  beyond  the  scope  of  evaluating  the  results  of  this 
source  selection  against  the  specific  issues. 

V.4.1  Summary  of  Issues  for  Which  ETIP's  Use  of  PFE 

Proved  Successful 

Overall  ETIP's  use  of  Parametric  Factor  Evaluation 
for  its  source  selection  for  the  two  contracts  was  success- 
ful. It  provided  for  a timely  award  without  protests  and 
it  was  able  to  deal  effectively  with  many  of  the  issues 
which  were  raised  by  participants  or  described  in  the  liter- 
ature on  source  selection.  ' ETIP  was  successful  in  taking 
a process  which  had  been  developed  and  used  in  a different 
organization,  for  the  procurement  of  a different  type  of 
system,  and  adapting  it  for  its  own  use. 

The  following  is  a listing  of  the  specific  issues  for 
which  ETIP's  use  of  PFE  proved  successful  (with  the  section- 
numbers  where  these  issues  were  discussed); 

o The  structure  enabled  bidders  to  demonstrate 
their  ability  to  deal  with  the  critical  aspects 
of  the  projects  (V.3.2.1). 

o The  structure  did  not  create  critical  gaps 

in  evaluating  bidders'  capabilities  (V.3.2.1). 

o Information  in  the  proposals  was  readily  lo- 
cated by  the  panel  when  it  was  needed  (V.3.2.1). 

o The  interrelated  documents  eliminated  contradic- 
tions in  the  RFP  (V.3.2.1). 

o Goldplating  of  proposals  was  not  a problem, 
the  proposals  were  not  too  large  (V.3.2.2). 
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o Bidders  were  confident  that  they  had  communi- 
cated their  qualifications  to  the  evaluation 
panel  ( V . 3 . 2 . 3 ) • 

o ETIP  was  able  to  deal  with  differences  in  uncer- 
tainty among  bidders  (V.3.2.4). 

0 Bidders  were  confident  that  they  had  related 
sub-factor  issues  to  the  objectives  of  the 
projects  ( V . 3 . 2 . 5 ) . 

o The  basis  for  competitive  evaluation  matched 
with  the  subsequent  work  requirements  of  the 
projects  (V.3.2.5  and  V.3.4.3). 

o The  sub-factors  encompassed  most  of  the  issues 
which  would  be  critical  for  developing  the 
systems  ( V . 3 • 2 . 6 ) . 

o The  sub-factors  did  not  deal  with  unimportant 
issues  ( V . 3 . 2 . 6 ) . 

o Parroting  back  of  information  from  the  RFP 

to  the  exclusion  of  other  information  required 
for  evaluation  was  not  a problem  (V.3.2.7). 

o Enough  proposals  were  received  so  that  the 
panel  was  confident  in  the  scores  it  assigned 
(V.3.3.1). 

o Trivial  bids  were  not  received  (V.3.3.2). 

0 The  process  enabled  a strong  capability  for 
evaluating  management  issues  (V.3.4.1). 

0 The  process  enabled  a strong  basis  for  evaluat- 
ing technical  information  (V.3.4.1). 

o ETIP  was  able  to  obtain  the  participation  of 
experts  on  the  evaluation  panel  (V.3.5.1). 

o Members  of  factor  teams  were  able  to  arrive 
at  initial  ratings  of  bidders  prior  to  the 
team  metings  (V.3.6.1). 

0 Factor  teams  were  able  to  reach  agreement  on 
sub-factor  scores  for  each  bidder  without  vot- 
ing ( V . 3 . 6 . 2 ) . 
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o Members  of  the  factor  teams  were  able  to  see 
how  their  participation  affected  the  ratings 
(V.3.6.3) . 

o Members  of  the  factor  teams  were  informed  of 
how  the  ratings  and  other  inform.ation  would 
be  used  to  select  contractors  (V.3*6.3). 

o The  factor  teams  were  able  to  deal  effectively 
with  members  who  used  different  standards  for 
their  initial  ratings  (V.3-6.4). 

o The  time  to  agree  on  weightings  for  sub-factors 
within  the  factors  was  minimal  (V.3.6.5). 

o The  panel  was  able  to  evaluate  the  biases  of 
its  members  and  use  them  in  making  its  ratings 
(V.3.7. 1 ) . 

o The  process  kept  records  of  the  biases  of  its 
members  ( V . 3 • 7 • 1 ) • 

0 Panel  members  who  were  biased  for  or  aeainst 
specific  bidders  were  not  able  to  sway  the 
rest  of  the  team  (V.3.7.1). 

o All  proposals  were  evaluated  on  the  same  issues 
using  the  same  criteria  (V.3.7. 2). 

o The  ability  to  use  all  outside  information 

increased  the  accuracy  of  the  scores  (V.3.7. 3). 

0 The  flexible  statement  of  work  which  ETI?  wrote 
was  acceptable  to  the  contracting  office 
(V.3.8. 1 ) . 

0 The  final  scores  provided  for  a range  of  over- 
all technical  discrimination  among  bidders 
(V.3.9.1). 

o The  sub-factor  scores  provided  for  technical 
discrimination  among  bidders  on  specific  issues 
(V.3.9.2). 

o Technical  leveling  and  technical  transfusion 
were  not  problems  (V.3.9.3). 

o Undue  weight  was  not  given  to  bid  price  in 
the  final  evaluation  (V.3.9.4). 


o Bidders  were  able  to  understand  the  process 
which  was  used  to  evaluate  their  proposals 
(V. 3. 10.1). 

o It  was  possible  to  trace  the  results  from  the 
initial  individual  ratings  of  the  team  mem.bers 
to  the  final  award  decision  (V. 3. 10.2). 

o The  panel  followed  the  rules  which  were  de- 
scribed to  the  bidders  (V.3.10.3). 

o The  weightings  of  sub-factors  after  the  scores 
were  known  did  not  produce  inequities  (V. 3. 10.4). 

0 The  process  is  legal  under  the  Federal  Procure- 
ment Regulations  (V. 3.1 1.1). 

o The  contract  negotiator  was  confident  in  being 
able  to  defend  the  results  (V.3.11.2). 

0 The  contract  negotiator  accepted  the  contract- 
ing officer’s  technical  representative's  report 
on  the  panel’s  recommendations  without  request- 
ing revisions  (V.3.11.3). 

o The  contracts  were  awarded  in  a timely  manner 

which  met  ETIP’s  fiscal  year  deadline  and  brought 
no  protests  (V.3.11.4). 

o The  resources  required  by  ETIP  to  prepare  the 
RFP  and  evaluate  the  proposals  were  not  exces- 
sive ( V . 3 . 1 2 . 1 ) . 

o The  resources  required  by  ETIP  to  follow-up 
after  the  panel  meetings  were  not  excessive 
(V. 3. 12.4). 

o The  process  provided  for  sufficient  education 
of  the  panel  members  to  enable  them  to  under- 
stand their  role  (V. 3. 13.1). 

V.4.2  Summary  of  Issues  for  Which  ETIP’s  Use  of  PFE 

Proved  Unsuccessful 

There  were  several  issues  which  ETIP’s  use  of  PFE  did 
not  deal  with  successfully.  These  were: 
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o ETIP  was  not  able  to  make  measurements  which 
would  predict  the  bidder’s  future  capabilities 
on  some  issues.  This  was  particularly  true 
for  trying  to  assess  future  commitment  of  man- 
agement to  the  projects  (V.3.^.3). 

o The  RFP  was  not  successful  in  communicating 
that  the  process  would  be  more  difficult  to 
corrupt  than  most  other  processes  (V.3.T0.1). 

o ETIP's  use  of  PEE  did  not  provide  sufficient 
education  about  the  projects  to  government 
personnel  who  would  be  working  with  the  con- 
tractors (V. 3. 13.2). 

V.4.3  Summary  of  Issues  for  Which  the  Results  of  ETIP’s 

Use  of  PEE  are  Equivocal 

The  results  of  ETIP's  use  of  PEE  for  some  of  the  issues 
are  uncertain,  given  the  information  used  in  this  evalua- 
tion. These  issues  are: 
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o Whether  the  source  selection  process  did,  in 
fact,  anticipate  the  problems  which  actually 
occurred  during  the  projects  (V.3.4.3); 

o Whether  experts  made  a significant  contribution 
to  the  evaluation  (V.3.5.2); 

0 Whether  junior  panel  members  made  a significant 
contribution  to  the  evaluation  (V.3.6.6); 


o 


Whether  the  use  of  the 
illustrative  sub-items 
tual  statement  of  work 


statement  of  work  with 
in  addition  to  the  ac- 
was  confusing  (V.3.8.I); 
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o Whether  the  use  of  a flexible  statement  of 
work  caused  any  better  qualified  firms  to  not 
bid  (V.3.8.2);  and 

o Whether  responding  to  the  RFP  provided  educa- 
tion for  the  bidders  which  was  useful  in  their 
future  interactions  with  ETIP  (V. 3. 1^.2). 
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CHAPTER  VI 

BROADER  IMPLICATIONS 


VI . 1 Introduction 

This  chapter  presents  a discussion  of  the  implications 

of  this  research  which  go  beyond  the  specific  case  studied. 

Four  topics  which  are  reviewed  are 

o the  methodology  used  for  this  research; 

o comparison  of  ETIP's  results  to  prior  and  sub- 
sequent uses  of  the  PFE  process; 

o adaptability  of  the  PFE  process  to  other  tvpes 
of  problems;  and 

o Dossible  topics  for  additional  research  on 
PFE. 


VI . 2 The  Case  Study  Method  for  Exploratory  Research 

The  form  of  this  research  was  exploratory.  It  w 
the  initial  evaluation  of  the  use  of  a process  for  wh 
no  prior  research  had  been  published.  As  such,  it  wa 
portant  that  a thorough  description  of  the  process  be 
veloped  and  that  a relatively  comprehensive  set  of  cr 
for  the  evaluation  be  generated.  In  addition,  the  au 
wished  to  represent  the  criteria  in  a structure  which 
enable  a point  of  departure  for  future  research  on  PF 
for  future  comparisons  of  PFE  to  other  source  select! 
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techniques.  The  case  study  method  which  was  used  was  felt 
to  provide  a credible  means  for  doing  this. 

When  little  prior  information  is  available  about  a 
program  or  process,  a complete  description  of  how  it  really 
works  (as  contrasted  with  descriptions  of  how  it  should 
work)  is  an  important  antecedent  to  later  evaluatioji.  It 
provides  a context  (model)  for  exploring  the  nature  of  the 
relationships  among  important  variables.  In  this  research 
the  description  was  based  upon: 

0 background  on  the  PFE  process  from  both  histor- 
ical documents  and  personal  communications 
with  its  primary  developer; 

o participation  in  the  process  at  ETIP  by  the 
author  and  direct  access  to  records  of  the 
source  selection  of  the  two  contractors;  and 

0 interview  and  questionnaire  information  from 
other  participants  in  the  source  selection. 

Background  on  PFE  was  studied  to  insure  an  accurate  context 

for  describing  the  process.  Interviews  and  questionnaires 

were  used  to  obtain  information  on  undocumented  events. 

Extensive  documentation  of  sources  was  provided  with  the 

chronology. 

The  criteria  for  the  evaluation  were  generated  by  exam- 
ining the  critical  issues  which  ETIP  faced  during  its  source 
selection.  A list  of  issues  was  drawn  by  two  methods. 

One  was  by  examining  the  records  of  ETIP’s  use  of  the  PFE 
process  for  questions  which  were  raised  before,  during, 
and  after  the  source  selection  by  the  people  who  were  in- 


volved.  The  other  was  by  writing  down  issues  which  seemed 
applicable  to  ETIP's  situation,  which  were  identified  from 
the  literature  on  source  selection,  discussions  v;ith  others, 
or  reflection  on  the  problem.  Comprehensiveness  in  covering 
the  major  issues,  rather  than  indeoendence  among  them,  was 
the  objective.  The  issues  were  then  sorted  into  groupings 
of  related  issues  to  provide  a structure  for  discussion. 

A subset  of  these  issues  were  selected  for  examination 
based  upon  the  two  criteria  of:  1)  the  relative  importance 

of  the  issue  to  the  problem  of  source  selection;  and  2) 
the  availability  of  data  to  support  an  evaluation  of  the 
issue.  The  issues  treated  most  extensively  were  those  which 
were  considered  most  important  and  for  which  data  were  avail- 
able or  obtainable.  Where  additional  data  required  to  eval- 
uate an  important  issue  could  be  obtained  from  other  par- 
ticipants in  the  process  they  were  requested  through  ques- 
tionnaires. 

For  each  selected  issue,  a conclusion  was  reached  about 
whether  PFE  did  or  did  not  present  an  effective  solution 
or  whether  the  results  appeared  inconclusive  given  the  data 
available.  In  each  case  the  data  used  to  reach  the  conclu- 
sion were  described. 

The  unique  structure  of  issues  developed  for  this  re- 
search provides  a basis  for  departure  and  comparison  for 
further  research.  For  several  issues  which  were  identified. 


the  data  available  here  were  not  sufficient  to  enable  con 


fident  conclusions  about  the  effectiveness  of  PFE  in  dealing 
with  the  problems.  During  future  uses  of  PFE,  data  collec- 
tion schemes  which  are  instrumented  in  advance  may  enable 
such  conclusions. 

Comparisons  between  PFE  and  other  system  source  selec- 
tion methods  may  be  made  by  using  the  issue  structure. 
Effectiveness  of  various  source  selection  methods  may  be 
compared  if  evaluations  are  accomplished  by  using  largely 
the  same  set  of  issues.  Case  study  designs  which  facilitate 
departures  and  comparisons  add  to  both  the  research  and 
administrative  value  of  the  results.  This  case  study  is 
felt  to  be  an  important  contribution  to  research  methods 
on  source  selection  and  good  example  of  a credible  approach 
to  exploratory  research  in  general. 

VI . 3 Comparison  of  ETIP*s  Results  to  Other  Uses  of  the  PFE 
Process 

VI. 3.1  Other  Experiences 

There  have  been  both  prior  and  later  experiences  with 
using  PFE.  The  prior  experiences  were  the  original  uses 
of  PFE  by  the  Air  Force  to  procure  hardware  subsystems. 

The  later  experiences  were  the  subsequent  uses  of  PFE  by 


ETIP  (in  1977  and  1978)  to  develop  an  evaluation  capability 
for  its  regulatory  program. 

VI. 3. 2 Lessons  Learned  by  Comparison  to  the  Wright 

Field  Experiences 

VI. 3. 2.1  Departures . ETIP  made  several  departures 
from  the  original  Wright  Field  applications  of  the  PtE  pro- 
cess. These  were: 

o The  ETIP  RFP  was  largely  prepared  by  one  per- 
son. Others  with  a stake  in  the  outcom.e  of 
the  procurement  were  not  involved  in  selecting 
the  sub-factors. 

o Some  of  the  exoerts  who  participated  on  STIP’s 
factor  teams  had  no  expected  direct  relation- 
ship to  the  future  projects. 

o The  use  of  a written  question  form.  (Figure 
IV. 4)  was  substituted  for  holding  oral  clar- 
ifications prior  to  the  factor  tea.m  meetings. 

o A statement  of  work  with  illustrative  sub-item.s 
for  the  first  phase  of  the  contract  and  a corre- 
sponding illustrative  schedule  were  added  to 
the  ETIP  RFP. 

VI.  3 .2. 2 Importance  of  early  involvement.  A .m.ajor 
lesson  which  is  demonstrated  by  this  experience  is  the  im- 
portance of  getting  key  people  on  the  project  involved  early 
in  the  preparation  for  the  source  selection.  This  partici- 
pation is  critical  to  the  development  of  support  and  under- 
standing which  will  be  required  during  the  future  project. 

Time  considerations  precluded  the  active  involvement 
of  others  in  the  preparation  of  the  ETIP  RFP.  This  did 
not  effect  ETIP’s  ability  to  select  contractors.  However, 
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it  probably  did  affect  the  course  of  the  projects  after 
the  successful  bidders  were  under  contract.  Commitment 
of  many  key  people  toward  the  development  of  the  systems 
seemed  low.  The  level  of  understanding  or  agreement  on 
the  importance  of  many  of  the  sub-factor  issues  was  also 
low . 

Early  involvement  of  key  people  (those  who  have  a stake 
in  the  process  or  outcome  of  the  project)  in  selection  of 
the  sub-factors  would  have  provided  an  important  two-way 
education  process  between  those  most  directly  responsible 
for  manageing  the  contracts  and  others  who  were  less  direct- 
ly involved.  This  early  involvement  would  also  have  helped 
demonstrate  to  people  that  their  concerns  regarding  the 
contracts  would  be  dealt  with  in  a direct  manner,  by  includ- 
ing them  as  sub-factors  in  the  source  selection.  Commitment 
would  probably  have  been  greatly  enhanced  if  people  had 
realized  that  they  could  be  full  participants  in  the  decision- 
making process.  The  author  believes  that  the  background 
provided  and  added  commitment  obtained  through  early  involve- 
ment by  people  who  will  later  be  involved  in  the  project 
is  critical  to  future  development.  This  step  should  not 

be  compromised  in  future  uses  of  the  process. 

* 

VI. 3. 2. 3 Selection  of  experts.  Some  of  the  experts 
who  participated  on  the  factor  teams  were  people  who  were 
not  expected  to  have  any  involvement  in  the  development 


of  the  systems.  The  discussion  under  section  V.3.5  indi- 
cates that,  though  ETIP  was  quite  successful  in  obtaining 
the  participation  of  experts  in  the  evaluation,  reactions 
were  mixed  over  the  usefulness  of ^their  contribution.  The 
key  question  was  whether  the  people  had  enough  knowledge 
about  the  projects  to  make  their  inputs  worthwhile.  Also, 
because  they  would  not  be  involved  later,  there  was  no  incen- 
tive for  them  to  treat  the  evaluation  as  anything  m.ore  than 
an  academic  exercise.  The  author  believes  that  the  evalua- 
tion was  not  harmed  by  the  involvement  of  these  people, 
and  often  they  did  provide  additional  useful  perspective. 
However,  he  also  believes  that,  where  available,  experts 
should  be  drawn  from  the  set  of  people  who  will  have  to 
work  with  the  resulting  contractors. 

VI. 3.2. 4 Benefit  of  orals  prior  to  factor  team  m.eet- 
ings . In  section  V. 3.2.4  the  author  indicated  that  the 
question  form  was  only  of  minimum  usefulness.  The  burden 
of  the  clarification  process  fell  upon  the  meetings  with 
the  bidders  who  remained  competitive  following  the  factor 
team  meetings.  Though  this  process  was  effective  for  ob- 
taining clarifications,  it  seem.s  as  though  it  also  caused 
the  addition  of  the  appearance  of  some  potential  impropriety 
or  "sub ject iveness"  in  the  process.  The  author  believes 
that  this  problem  would  have  been  reduced  had  oral  clarifi- 
cations been  held  before  the  factor  team  meetings. 
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VI. 3.2.5  Problem  from  adding  an  illustrative  statement 
of  work.  The  addition  of  the  statement  of  work  with  illus- 
trative sub-items  for  the  first  phase  of  the  contract  seemed 
to  cause  confusion  after  the  contract  was  signed.  Both 
ETIP  and  contractor  personnel  began  the  projects  by  treating 
the  illustrative  sub-items  as  though  they  were  contractually 

t 

required.  This  was  a source  of  problems  since  it  diverted 
effort  from  the  major  objective  of  developing  the  evaluation 
systems. 

VI. 3. 2. 6 Transferability  of  the  process  to  a ”soft- 
ware”  problem.  A major  lesson,  which  was  documented  by 
this  case  study,  is  that  the  PFE  process  may  be  adapted 
for  use  from  the  procurement  of  "hardware”  systems  to  the 
procurement  of  "software"  systems.  Use  of  the  process  by 
ETIP  was  initiated  on  the  premise  that  the  underlying  prob- 
lems were  sufficiently  similar  (Thompson,  1976a,  p.  170). 

The  adaptation  was  clearly  successful. 

Two  points  which  stand  out  from  the  experience  of  using 
PFE  for  both  "hardware"  and  "software"  applications  are 
that:  1)  the  technical  evaluators  are  able  to  retain  re- 

sponsibility for  the  technical  recommendations;  and  2)  the 
resultant  selection  is  not  subject  to  successful  attack. 

Those  in  the  chain  of  command  are  unable  to  exercise  polit- 
ical control  over  the  selection.  The  thoroughness  and  trace- 
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ability  of  the  process  prevent  this.  These  same  attributes 
of  the  process  make  it  clear  that  the  selection  will  with- 
stand any  protest  investigation. 

VI. 3*3  Modifications  Made  in  Later  Uses  of  PFE  by 

ETIP 

The  factors  and  sub-factors  which  were  used  for  the 
later  ETIP  source  selections  were  essentially  the  same  as 
those  used  i'n  RFP  for  the  procurement  evaluation  systems. 

They  were  considered  to  be  both  the  essential  issues  and 
relatively  complete.  However,  there  were  some  additions 
to  the  illustrative  questions  under  the  sub-factors. 

Some  changes  were  made  in  the  way  the  process  was  imple- 
mented. There  was  earlier  and  more  extensive  involvem.ent 
by  those  who  would  be  working  on  the  projects  (both  within 
and  outside  of  ETIP).  The  schedules  of  factor  team  m.eetings 
were  distributed  a week  or  two  in  advance  of  the  closing 
date  for  proposals,  enabling  panel  members  to  adjust  their 
appointments  to  the  factor  team  meeting  tim.es  rather  than 
vice  versa.  Finally,  references  which  bidders  listed  in 
their  proposals  were  called  in  advance  of  the  m.anagement 
factor  team  meetings,  so  that  this  information  could  be 
considered  in  assigning  ratings.  All  of  these  m.odif ications 
from  ETIP’s  initial  use  of  PFE  seemed. to  be  improvements. 
Others  contemplating  use  of  the  PFE  process  in  the  procure- 
ment of  evaluation  systems  may  wish  to  consider  additional 


modifications . 
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VI . 4 Adaptability  of  the  PFE  Process  to  Other  Types  of  Prob- 
lems 


Parts  of  the  process  of  PFE  may  be  useful  for  problems 


other  than  goyernment  source  selection.  Though 
tion  required  for  portability  to  other  problems 
been  proyided  here,  it  is  worth  mentioning  what 
possible  applications  may  be.  Thompson  (1976a) 
lated  about  types  of  problems  for  which  the  PFE 
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might  proye  useful.  He  has  written  that: 

A number  of  possible  applications  might  be  sug- 
gested, and  the  categories  which  follow,  and  the 
illustrations,  represent  only  one  way  in  which 
they  might  be  organized.  First,  the  case  of  multi- 
ple candidates  (bidders  or  projects)  which  must 
be  compared  against  an  uncertain  or  complex  set 
of  requirements,  including: 

a)  competitiye  procurem.ent  based  on  a complex 
set  of  design  requirements  or  of  a system, 
e.g.,  the  present  case; 

b)  rank  ordering  of  a series  of  competing 
projects  (which  may  not  be  directly  compar- 
able) against  the  twin  standards  of  likeli- 
hood of  success  and  probable  payoff  (future 
yalue),  i.e.,  the  "project  selection"  pro- 
cess ; 

c)  comparison  of  alternatiye  techniques,  solu- 
tions, or  (assumed)  resources  and  con- 
straints against  a set  of  "independent" 
and/or  differentially  affected  requirements, 
e.g.,  cost-benefit  analysis. 


Second,  the  case  of  a single  Candidate  (bidder 
or  project)  which  must  be  compared  against  a com- 
plex or  uncertain  set  of  requirements,  including: 

a)  a project  or  proposal  submitted  for  "accep- 
tance" under  a complex  statutory  require- 
ment, e.g.,  application  for  qualification 


as  a health  maintenance  organization  (HMO), 
submission  of  an  ’’impact  statement"  for 
a power  plant  or  other  facility; 

b)  comparison  of  a prooosed  oroject  against 
a set  of  "independent"  requirements,  e.g., 
cost-benefit  analysis. 

Third,  the  case  where  one  (or  more)  candidates 
are  to  be  compared  against  one  (or  m.ore)  uncertain 
or  complex  requirements  by  a set  of  raters  who 
may  differ  not  only  in  the  kind  of  infor.mation 
they  will  accept  but  also  in  their  value  struc- 
ture; in  at  least  some  cases  where  group  decision 
processes  are  used  (e.g.,  arbitration,  negotia- 
tion, the  Delphi  Method,  and  perhaps,  some  politi- 
cal processes)  it  may  be  desirable  to  initially 
separate  out  those  areas  which  are  "value-free," 
i.e.,  facts  or  agreed  statements  of  facts,  to 
allow  a more  narrow  concentration  on  those  rem.ain- 
ing  areas  in  which  there  are  value-based  differ- 
ences (Thompson,  1976a,  pp.  179-lSO). 

These  types  of  problems  occur  regularlv  in  both  govern- 
ment and  industry  and  are  often  resolved  in  a less  than 
successful  manner.  Where  current  practice  is  not  expected 
to  produce  satisfactory  results,  experimentation  with  alter- 
native techniques,  such  as  PFE,  seems  desirable. 


VI . 5 Possible  Additional  Research 

Several  areas  of  further  research  may  produce  addition- 
al useful  information  about  the  effectiveness  of  the  PFE 
process  or  may  suggest  modifications  for  improving  the  pro- 
cess. 

This  case  study  focused  primarily  on  evaluating  the 
effectiveness  of  PFE  in  solving  problems  during  source  selec- 
tion. It  did  not  follow  the  projects  past  the  signing  of 
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the  contracts  to  try  to  determine  the  match  between  ratings 
on  sub-factors  and  subsequent  performance.  In  the  future, 
where  it  can  be  instrumented  in  advance,  it  may  be  possible 
to  try  to  obtain  time  series  measurements  on  the  performance 
of  successful  bidders  on  each  sub-factor.  It  may  also  be 
possible  to  keep  track  of  problems  which  occur  during  the 
projects  and  check  to  see  whether  or  not  th'ey  were  antici- 
pated by  the  source  selection  process. 

Research  directed  toward  improving  PFS  may  focus  on 
determining  the  sensitivity  of  the  group  decision-making 
process  to  changes  in  various  independent  variables.  Such 
things  as  influence  structures,  order  of  discussion  of  bid- 
ders, use  of  oral  clarifications  prior  to  factor  team  meet- 
ings, and  variations  in  group  size  all  seem  to  be  fruitful 
areas  for  further  investigation. 

VI . 6 Summary 

This  chapter  reviewed  implications  of  this  case  study 
which  go  beyond  the  evaluation  of  the  specific  use  of  PFE 
for  the  selection  of  contractors  for  ETIP’s  procurement 
evaluation  systems.  It  reviewed  the  methodology  which  was 
used  for  the  study,  the  increases  in  knowledge  about  the 
PFE  technique  gained  from  comparison  to  prior  and  subse- 
quent uses  of  the  process,  and  the  potential  portability 
of  the  PFE  process  to  other  problems.  Finally,  suggestions 
were  made  for  areas  for  future  related  research  on  the  pro- 


cess . 
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I.  INTRODUCTION 


The  objective  of  the  research  v/hich  culminated  in  this  set  of 
documents  i-^as  to  develop  a process  for  the  procurement  of  two 
evaluation  systems.  IVithin  the  context  of  applicable  procurement 
regulations  and  program  plans  , the  set  provides  a complete  and 
integrated  framework  for  both  the  preparation  of  proposals  by 
prospective  bidders  and  the  evaluation  of  proposals  by  the  eval- 
uation team. 

The  document  entitled  Proposal  Preparation  Procedure  provides 
both  an  overall  description  of  the  process  and  specific  guidance 
to  prospective  bidders.  All  of  the  salient  or  significant 
characteristics  of  the  system  to  be  procured,  the  program  under 
which  it  will  be  developed,  and  supporting  elements  are  included 
in  a set  of  elements  called  sub-factors  which  are  grouped,  for 
convenience,  into  a set  of  factors.  Each  of  the  key  documents -- 
the  Statement  of  V/ork,  Schedule,  Scope  of  Effort,  and  Evaluation 
Factors - -follows  a parallel  organization  of  the  set  of  sub-factors 
It  is  this  integration  of  all  of  the  central  documents  which 
provides  both  proposer  and  evaluator  with  a single,  common  struc- 
ture . 
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The  initiation  of  the  development  of  this  process  arose  out 
of  the  recognition  that  the  ''evaluation  systems"  to  be  procured 
required  the  services  of  a systems  contractor  and  that  it  appear- 
ed that  neither  ETIP  nor  the  "evaluation  industry"  had  any 
significant  present  capability  to  design  or  m.anage  such  a system. 

A review  of  available  proposal  preparation  and  evaluation  pro- 
cedures suggested  that  some  further  development  was  required. 

The  process  described  herein  was  used  to  procure  the  tv;o 
systems,  and,  as  of  the  initial  getting  started  period  under  the 
two  contracts,  appears  to  have  worked  out  reasonably  well  in  terms 
of  our  confidence  that  we  have  chosen  not  only  the  best  but  also 
reasonably  competent  contractors. 

While  this  set  of  documents  was  specifically  designed  for  the 
procurement  of  these  two  evaluation  systems,  the  process,  with 
modifications  in  the  set  of  sub-factors,  is  applicable  to  compar- 
able systems  problems  involving  complex  or  uncertain  requirements. 
One  of  the  purposes  of  this  report  is  to  make  these  documents 
available  on  the  basis  that  others  may  be  interested  in  further 
applications. 
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Introductory  Notes 

The  purpose  of  these  two  proposed  procureraents  is  to  obtain 
contractor  assistance  in  planning,  designing,  and  carrying 
out  the  evaluation  of  a series  of  procurenent  experin'.ents 
which  are  being  undertaken  by  ETIP  in  cooperation  v/ith  several 
governmental  procurement  agencies. 

The  set  of  objectives  of  the  evaluation  program  includes  the 
following : 

a.  an  overall  description  of  the  experiments  as  well  as 
selected  detailed  descriptions 

b.  an  overall  assessm.ent  of  both  the  immediate  and  the 
subsequent  effects  (impacts)  of  the  experiments,  as 
well  as  selected  detailed  assessments 

c.  An  evaluation  system  or  process  v/hich  can  be  used  by 
the  appropriate  governm.ent  agency  (ies)  to  obtain 
evaluations  on  a continuing  basis  of  these  as  v;ell 
as  future,  similar  experiments. 

There  are  a number  of  choices  v/hich  could  be  made  in  obtaining 
contractor  assistance,  ranging  from  a single  overall  contractor 
to  individual  contractors  for  each  experiment,  or  co.mbinations 
in  between.  The  choice  v;hich  has  been  made  is  to  divide  the 
overall  evaluation  program  into  tv/o  procuremLents , as  follows: 

a.  an  evaluation  of  "agency  impact"  v/hich  will  direct 
its  attention  to  a description  of  the  experim.ents , 

an  assessment  of  the  effects  on  the  several  governm.ent 
agencies,  and  the  design  of  the  related  evaluaoicn 
system.  "Agency  impact"  includes  both  immediate  and 
subsequent  effects  on  the  program  administrative  agency 
but  also  other  affected  or  related  governm.ent  agencies 

b.  an  evaluation  of  "commercial  impact"  v/hich  v/ill  direct 
its  attention  to  the  immediate  and  subsequent  effecrs 
upon  the  commercial  (or  industrial)  sector,  and,  as 
appropriate,  consumers  or  users  of  the  products,  and 
the  design  of  the  related  evaluation  system. 


1 


RFP  6-35756 


361 


The  set  of  "experiments”  includes  the  following: 

a.  specific  procurement  experiments  - including 
"completed,"  on-going,  and  planned 

b.  specific  related  or  supportive  experiments  - includ- 
ing procedures  for  selecting  experim.ents , training 
of  agency  personnel,  processes  for  obtaining  from 

or  providing  to  the  industrial  sector  information 
required  for  carrying  out  procurement  experiments, 
and  other  administrative  changes 

c.  general  "experimental"  changes  - related  to  or  in 
support  of  the  above. 

The  set  of  experiments  may  be  conveniently  grouped  into  broad 
categories  according  to  these  dimensions: 

a.  by  program  administrative  agency  - CD  Federal 

Supply  Services  (FS3) ; (2)  state  and  local  procure- 

ment agencies  (S&L) ; and  (3)  Veterans  Administration 
(VA) 

b.  by  type  of  procurement  experim.ent  (1)  life  cycle 

costing  (LCC) ; (2)  value  incentive  contracting  (VIC) ; 

(3)  performance  specifications;  (4)  multi-year  awards; 
etc . 

c.  by  the  kind  of  effect  intended  - (1)  economic  or  per- 
formance advantage  to  the  using  government  agency; 

(2)  related  or  derivative  improvement  in  procurement 
capabilities  of  the  program  administrative  agency; 

(3)  increased  or  improved  technological  innovation 

in  the  industrial  sector;  and  (4)  economic  or  perfor- 
mance advantage  to  civilian  sector  users. 

To  obtain  the  evaluation  desired  we  propose  to  let  two  contracts, 
each  for  a three-year  period,  divided  into  three  phases,  as 
follows: 

Phase  One  - to  include,  generally,  three  activities 

(a)  preliminary  systems  analysis 

(b)  evaluation  of  specific,  selected  procurement  or 
other  experiments 

(c)  design  and  pilot  test  of  the  related  evaluation 
system 
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Phase  Tv/o  - to  include,  generally: 

(a)  refinement  of  (a)  above 

(b)  evaluation  of  additional  specific  procurement 
or  other  experiments 

(c)  refinement  and  prototype  test  of  the  related 
evaluation  system 

Phase  Three  - to  include,  generally,  "turn  key"  implementa- 
tion of  the  related  evaluation  system  by 
the  appropriate  governm.ent  agency 

The  complete  procurement  package  has  been  designed  to  assure  the 
fullest  possible  interchange  of  information  betv;een  prospecrive 
bidders  and  those  responsible  for  the  evaluation  of  proposals. 
Because  of  this  it  v/ill  be  necessary  to  examine  the  complere 
set;  particular  attention  is  directed  to  the  fact  than  the 
Statement  of  Work  and  the  Schedule  must  be  read  in  conjunction 
with  the  accompanying  documents. 
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Some  Definitions 


Note:  In  most  cases,  the  context,  or  reference  to  other  parts 

of  the  bid  set  and  associated  documents,  should  resolve 
ambiguities  or  otherwise  clarify  definitions  of  terms.  These 
"definitions"  are  provided  as  a convenience  in  reference. 

ETIP  - Refers  to  the  Experimental  Technology  Incentives  Pro- 
gram. In  context,  the  reference  may  be  to  the  objectives  or 
content  of  the  program,  to  the  office  or  organization  itself 
(e.g.,  as  contracting  agency),  or  to  individual  staff  members. 

PAA  - Refers  to  the  Program  Administrative  Agency  (or  Agencies) . 
There  are,  for  these  procurements  (agency  impact  and  comjnercial 
impact),  three  PAA,  as  follows:  1)  Federal  Supply  Service 

(FSS) ; 2)  state  and  local  procurement  agencies  (SSL) ; 

3)  Veterans'  Administration  (VA) . .In  context,  the  reference 
v/ill  usually  be  to  the  specific  procurem^ent  office  (s)  and/or 
responsible  personnel  in  the  agency  which  have  the  direct 
authority  and/or.  responsibility  for  the  decision  to  intro- 
duce the  specific  experimental  intervention  and,  usually, 
for  the  conduct  of  the  program  and/or  function  to  which  the 
experimental  condition  is  applied.  For  FSS,  this  may  include 
both  central  and  regional  procurement  functions  or  offices. 

For  SSL,  this  may  include  NASPO,  NIGP,  and/or  specific  state 
and  local  procurement  agencies.  For  VA  this  may  include  both 
central  and  regional  or  local  procurement  agencies. 

AGENCY  IMPACT  - Refers  to  effects  or  impacts  on  or  in  the  PAA 
and,  where  applicable,  closely  related  governmental  agencies, 
and,  particularly,  agencies  which  are  the  users  of  the  products 
procured.  These  impacts  or  effects  include  not  only  those 
which  are  the  specific  objectives  of  a specific  or  general 
experiment,  but  also  significant  other  effects,  such  as 
changes  in  administrative  policies  and  procedures . 

COMMERCIAL  IMPACT  - Refers  to  effects  upon  the  commercial 
sector  and/or  civilian  sector  users.  These  impacts  or  effects 
include  not  only  those  which  are  the  specific  objectives  of  a 
specific  or  general  experiment,  but  also  significant  other 
effects,  such  as  increased  or  earlier  use  of  new  technology. 

EXPERI14ENT  - Refers,  generically,  to  any  combination  of 
intervention  and  desired  (or  hypothesized)  effect.  .In  a 
complex  of  interventions  and  effects,  there  may  be  a number 
of  "experiments,"  including  experiments  which  are  combina- 
tions of  other  experiments,  and  experiments  where  the  effect 
of  one  is  the  intervention  of  another. 
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SPECIFIC  EXPERIMENT  - Refers,  usually,  to  a specific,  identified 
intervention  and  desired  (or  hypothesized)  effect.  The  major, 
and  most  significant,  cases  are  the  specific  procurement  experi- 
ments (e.g.,  use  of  LCC  to  procure  air  conditioners).  T.here 
may  be  other  (related)  specific  experim.ents  (e.g.,  the  FSS  train- 
ing program) . 

VARIABLE  - Refers,  generically,  to  any  object,  event,  state, 
value,  function,  etc.  In  the  context  of  a specific  experiir.e.nt 
(or  study),  the  independent  variable  is,  usually,  the  i.nter- 
vention,  and  the  dependent  variable  the  desired  (or  hypothesized) 
effect . 

PAPJi^lETER  - Refers  to  those  variables  v;hich,  whether  controllable 
(directly  or  indirectly)  or  not,  affect  (or  are  hypothesized 
to  affect)  plausibly  and/or  significantly  uhe  inrerprerability 
and/or  credibility  of  the  observed  (or  hypothesized)  relation- 
ship betv;een  the  variables  in  a specific  experimeno . 

A PRIORI  PROPOSITION  TESTING  - Refers  to  those  specific  experi- 
ments for  v/hich  one  of,  if  not  the  prim.ary,  purposes  is  to 
establish,  with  a relatively  high  degree  of  credibility,  the 
relationship  betv/een  the  interve.ntion  and  the  effeco  observed. 

In  practice,  this  will  be  largely  limited  to  relatively  v;ell 
defined  interventions  and  proximal  effects,  and  vrhere 
reasonable  control  of  paramieters  can  be  achieved. 

EXPLORATORY  (AND/OR  DESCRIPTIVE)  - Refers  to  experim.ents  (and/ 
or  studies)  where  it  is  either  not  feasible  or  not:  required 
to  meet  the  requirem.ents  of  a priori  proposition  testing.  In 
practice,  this  will  apply  to  requirements  to  describe  and/or 
measure  sets  of  variables  v.dnere  either  the  description  is 
sufficient  itself  or  provides  a basis  for  identifying  hypotheses, 
variables,  and/or  parameters. 

EVALUATION  SYSTEM  - Refers  to  the  set  of  policies  and  procedures 
which  provides  the  basis  for  evaluating  the  (agency  and/or 
commercial)  impacts  of  a set  of  specific  procurem.ent  or  related 
experiments.  The  form,  detail  and  completeness  of  the  system, 
will  progressively  change.  In  Phase  One,  it  may  begin  as  a 
preliminary  outline  within  which  detailed  "single  thread" 
designs  are  developed  for  the  early  evaluation  of  specific 
procurement  experiments;  later,  the  results  of  the  preliminary 
systems  analysis  and  the  "pilot  test"  will  provide  the  basis 
for  a preliminary  evaluation  sysrems  design.  In  Phase  Two, 
progressive  refinement  should  result  in  a relatively  com.plete 
evaluation  system  v/hich  can  be  tested  as  a prototype,  in  part, 
through  the  evaluation  of  specific  expt=rim.ents  . In  Phase 
Three,  the  evaluation  system,  should  be  in  the  form,  of  a stable 
and  complete  system. 
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BASIC  (OR  BASE-LINE)  EVALUATION  PROCESS  - Refers  to  that  part 
of  the  evaluation  system  which  includes  those  evaluation 
activities  which  deal  with  key,  predictable,  identifiable, 
continuing,  common  objectives  for  v/hich  the  process  can  be 
institutionalized . 

SPECIAL  EVALUATION  PROCESS  - Refers  to  those  evaluation  activi- 
ties which  do  not  meet  the  above  requirements  because  of 
specialized,  one-time  or  changing  objectives. 

FRAJ4EW0RK  OR  MODEL  - Refers  to  graphical  or  other  conceptual 
representation  of  the  set  of  interrelations  among  objectives, 
organizations,  experiments,  and/or  programs,  and  the  evalua- 
tion system. 
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PROPOSAL  PREPAPJ\.TIOM  PROCEDURE 


A.  Purpose 

The  purpose  of  this  docuraent  is  to  provide  a guide  for 
bidders  in  the  preparation  of  their  proposals. 

B.  Basis  for  Preparation  of  This  Document 
B.l  Historical  Note 

A v/ide  variety  of  methods  have  been  used  for  the  procurement 
of  the  services  of  evaluation  contractors,  including  uhe 
following:  1)  adding  evaluation  as  an  express  or  im.plied 

requirement  to  the  basic  contract  for  the  program  or  exueri- 
ment;  2)  contracting  separately  with  a contractor  or  consuluan 
for  the  evaluation;  3)  obtaining  an  evaluation,  usually  post 
hoc / as  part  of  the  study  phase  of  a ccnrract  for  a new  or 
different  program.  The  description  of  the  services  desired 
may  take  many  forms,  including  the  following:  1)  a brief, 

general  requiremient  "to  perform  an  evaluation";  2)  stare- 
raents  or  descriptions  of  various  lengths  and  v;ith  various 
degrees  of  detail  and  comiuleteness  v;hich  outline  the  object- 
ives of  the  program  and/or  the  specific  questions  to  be 
answered;  3)  relatively  detailed  and  complete  " specif icaticns " 
of  the  services  required,  including  instrum^ents  to  be  used, 
sampling  plans,  and  the  form  of  the  analysis  required. 

In  some  cases,  and  for  some  purposes,  these  m.ethods  provide 
a satisfactory  base  for  defining  the  services  required, 
which,  in  turn,  may  provide  a satisfactory  basis  for  deoerm- 
ining  the  qualifications  of  prospective  contractors . 

None  of  the  above  appear  appropriate  as  a basis  for  determin- 
ing the  qualifications  of  prospective  ccntracoors  on  this 
procurement.  For  this  reason  this  docu.ment  was  prepared, 
using  a "parametric  factor  analytic  approach"  which  v;as 
developed  for  and  used  on  a number  of  subsystem,  procurem.ents 
by  another  agency  of  the  governm.ent. 
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B.2  Basis  for  Evaluation  (of  Bidders) 

Generally/  evaluation  (including  proposals)  requires  three 
things:  first,  a standard  or  base  for  comparing;  second, 

information  to  be  compared  against  the  standard;  and,  third, 
an  effective  method  for  doing  the  comparing.  These  are 
discussed  in  the  three  paragraphs  below. 

B.2.1  The  first  requirement  of  the  evaluation  of  competitive 
proposals  is  a standard  or  base  for  comparing,  and  this  is, 
obviously  "v;ho  can  do  the  job  best."  but  this  is  a matter  of 
predicting  because  it  is  a future  matter,  and  there  is 
uncertainty  in  specifying  the  job.  The  choice  of  standard 
for  selection  includes: 

a.  The  v/hole  program,  either  as  it  is  or  as  it  should 
be  stated  to  obtain  the  selection  base. 

b.  The  "significant  independent  variable"  - if  one 
part  will  determine,  either  by  its  absence  or 

. presence,  the  one  bidder  who  can  do  it,  then 

this  is  the  basis. 

c.  Some  sample  of  parameters  (variables  or  "factors") 
which  on  a presence  or  absence  basis,  or  qualitative 
scale,  is  the  determinant. 

The  choice  is  between  b)  and  c)  because  of  the  inability  to 
precisely  define  the  skills  and  content  of  the  future 
completely;  and  because  agreement  on  b)  presumes  considerable 
confidence  in  past  experience,  the  basis  must  be  c) . The 
development  of  c)  is  summarized  in  the  paragraph  below  in 
terms  of  a brief  statement  of  the  program. 

The  nature  of  the  item  to  be  procured  determines  the  program. 

The  evaluation  here  is  not  limited  to  fully  designed  evalua- 
tions of  specific  experiments,  nor  is  the  alternative  of  a 
broad,  overall  descriptive  case  study  sufficient.  It  is 
proposed  to  obtain  not  only  preliminary  systems  study  which 
will  include  both  of  the  above,  but  also  both  kinds  of 
evaluations;  in  subsequent  phases,  it  is  proposed  to 
develop  and  test  an  ongoing  capability  to  carry  out  such 
evaluations  in  such  form  and  detail  as  will  allov;  the 
government  to  implement  subsequent  phases.  Direct 
description  of  such  a program  must  necessarily  be  accomplished 
by  selecting  and  tabulating  the  significant  parts  of  the 
program;  and  these  parts  may  then  be  further  defined  and 
amplified  by  supporting  data.  These  parts,  identified  as 
"factors,"  are  the  bases  for  the  establishment  of  the 
standard;  and,  in  turn,  the  information  to  be  furnished 
and  the  evaluation  procedures. 
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B.2.2  The  information  to  be  compared  against  the  standard 
is  that  furnished  by,  or  about,  the  bidders.  This  requires 
cooperation,  varying  from  a little  to  a lot.  Too  little 
occurs  v/hen  the  bidder's  data  is  absent  or  cannot  be  trans- 
lated into  a comm.on  dimension  or  is  irreleva.nt.  Too  much 
occurs  v/hen  the  bidder  essentially  repeats  back  the  guide- 
lines furnished.  The  problem  of  "too  m.uch"  has  not  been 
experie.nced  in  prior  evaluations,  and  those  fev/  cases  of 
"parroting"  were  obvious.  The  concern  is  to  assure  that 
the  bidder  has  thorough  guidance  in  preparing  the  necessary 
information,  and  by  this  means  the  bidder  v/ill  be  aided 
in  directing  his  efforts  more  efficiently. 

B.2.3  Because  of  the  large  numl^er  of  factors  to  be  evalu- 
ated, the  large  volume  of  data  to  be  considered,  a.nd  the 
size  of  the  evaluation  team,  the  process  needs  to  be  v;ell 
organized  and  supported  with  efficient  a.nd  clear  procedures. 
Forms  and  instructions  must  be  prepared,  a.nd  arra.ngements 
for  scheduling  m^eeti.ngs  and  for  carrying  out  the  evalua- 
tion have  to  be  thought  out  in  advance.  A corollary  benefit 
is  an  increase  in  the  assurance  of  impartiality  through- 
avoidance  of  the  confusion  of  setting  rules  and  judging 
at  the  same  time. 

B. 3 Preparation 

This  document,  and  the  associated  statement  of  v/ork,  v/as 
prepared  upon  the  above  basis.  As  source  m.aterial, 
procedures  similar  to  this,  as  well  as  a number  of 
other  evaluation  procedures,  were  reviewed. 
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C.  General  Coinments  on  Proposal  Evaluation  Procedure 

To  provide  bidders  with  a general  background  and  perspective, 
a brief  outline  of  the  overall  procedure  is  furnished  belov;. 

(It  should  be  noted  that  these  comments  are  intended  only  as 
a general  description,  and  some  changes  may  be  made.) 

C.l  Preparation  for  Proposal  Evaluation 

C.1.1  To  assure  continuity,  personnel  responsible  for  setting 
up  and  administering  the  program  for  which  proposals  are  being 
solicited,  will  conduct  the  proposal  evaluation,  v/ith  the 
advice  and  assistance  of  others  who  are  less  directly  con- 
cerned. Direct  management  responsibility  is  centered  in  ETIP 
in  the  Contracting  Officer’s  Technical  Representative  (COTR) . 
He,  together  with  other  members  of  the  ETIP  staff,  v;ill  be 
working  closely  with  key  members  of  the  staff  of  the  program 
administrative  agency  in  the  type  of  close  relationship 
which  is  characteristic  of  "administrative  experimentation." 

As  a minimum,  it  is  intended  that  at  least  two  members  of 
the  proposal  evaluation  team  will  be  assigned  prim.ary  re- 
sponsibility for  each  subfactor,  with  backup  assistance  in 
review  by  other  team  members.  These  evaluation  team  mem±)ers 
are  responsible  for  preparing  and/or  reviewing  the  bases 
for  the  evaluation  and  for  briefing  the  bidders. 

C.1.2  The  results  of  the  preparation  are  incorporated, 
primarily,  in  the  RFP.  In  addition  to  the  Form  33  and 
accompanying  provisions,  the  PRP  includes  a series  of 
related  documents  prepared  for  the  primary  purpose  of 
furnishing  a base  for  the  preparation  of  proposals.  The 
necessary  clarification  and  revision  to  provide  a contract- 
ual basis  will  be  accomplished  by  negotiation  with  the 
selected  contractor.  Briefly,  these  documents  are 
grouped  as  follows: 

Statement  of  Work  - tabulates  the  significant  parts 
of  the  program  with  clarification  limited  primarily 
to  a brief  summary.  This  v;ill  include  those  separate- 
ly identifiable  "deliverables"  and  a schedule. 

Proposal  Preparation  Procedure  (this  document)  - provides 
an  explanation  of  the  basis  for  proposal  preparation. 

Proposal  Evaluation  Factors  - provides  the  basis  for 
proposal  preparation. 

Scope  of  Effort  - cost  proposal  requirements 

Other  Documents  - development  plans,  reports,  references 
etc.,  to  assist  the  bidders  in  preparing  their  proposals. 
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C.  1.2.1  To  the  extent  possible,  all  of  these  documents  follov; 
the  same  order.  The  Statement  of  V.’ork  includes  ite.ms  of 
required  v;ork  divided  into  three  phases.  The  Proposal 
Preparation  Procedure  is  designed  to  match  the  organization 
of  the  Statement  of  VJork.  It  is  intended  to  provide,  v;here 
necessary,  cross-reference  to  the  other  documents. 

C.1.2.2  To  assure  that  all  bidders  have  complete  sets  of  the 
related  documents,  and  especially  those  included  under 
"Others"  which  may  be  furnished  at  the  briefing,  a complete 
index  of  all  documents  v;ill  also  be  furnished  during  the 
briefings,  as  required. 

C.1.3  It  is  planned  to  brief  the  bidders  twice.  At  the 
initial  briefing,  a summary  presentation  on  the  progra.m 
will  be  given,  including  a presentation  by  the  program 
administrative  agency  on  the  administrative  objectives 
for  the  program.  (To  assure  the  continuity  specified  above, 
representatives  of  the  program  adm.inistrative  agency  v/ill 
participate  in  the  evaluation.)  Following  the  presentation, 
an  extensive  question  period  v;ill  be  scheduled.  The  second 
briefing  will  follov;  a fev;  v;eeks  later  after  the  bidders 
have  had  an  opportunity  to  study  the  program;  this  is  prim.arily 
to  allow  for  further  questioning  by  the  bidders  to  correct 
oversights  and  resolve  ambiguities.  No  inf or.matio.n  v;ill 
be  available  during  the  proposal  preparation  period  on  an 
individual  basis. 

C.2  Proposed  Evaluation  Procedure 

C.2.1  After  receipt  of  the  v;ritten  proposals,  the  proposal 
evaluation  team  will  have  a period  of  a few  days  to 
familiarize  themselves  generally  with  the  proposals;  and, 
specifically,  the  parts  within  their  responsibility.  This 
is  primarily  to  furnish  a basis  for  the  next  stage. 

C.2. 2 According  to  a schedule  to  be  established  at  the  second 
of  the  two  briefings  noted  above,  each  bidder  will  be  invited 
to  make  an  oral  presentation,  approxim.ately  one  hour  in 
length,  following  which  the  evaluation  team  will  have  two  to 
four  hours  for  questioning.  Purpose  of  the  questioning  is 
to  assure  that  the  evaluation  team,  understands  the  proposals , 
and  has  an  adequate  basis  for  evaluation  on  each  sub-factor. 
Both  weak  and  strong  points  v;ill  be  explored,  and  check  lists 
will  be  recorded  to  assure  that  team  memJDers  may  have  available 
to  them  both  the  written  and  oral  presentations . 
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C.2.3  Following  the  above  will  come  a further  period  of 
review,  during  which  individual  team  members  will  complete 
their  individual  evaluations  of  sub-factors  for  which  they 
are  responsible  or  concerned.  These  individual  evaluations 
v;ill  be  noted  with  the  assistance  of  a check  list  based  on 
a value  rating  scale  to  avoid  the  confusion  caused  by 
numerical  rating. 

C.2.4  Members  of  the  proposal  evaluation  team  concerned  with 
each  factor  will  meet  and  perform  a joint  evaluation.  They 
will  follow  an  established  procedure  entitled,  "Factor  Team 
Guidelines,"  to  assure  an  orderly  and  efficient  process. 

Briefly,  they  will  first  compare  notes  informally  to  assure 
common  standards  and  information;  second,  they  v/ill  arrange 
the  bidders  in  descending  order;  third,  they  vzill  establish 
the  relative  spread  among  the  bidders;  and,  finally,  their 
result  v;ill  be  permanently  recorded  and  signed  by  all  team 
members  present.  This  record  becomes  part  of  the  permanent 
evaluation  files.  Generally,  they  will  rate  on  each  sub- 
factor, as  applicable,  three  central  points: 

a.  Comprehension  or  understanding  of  the  problem 
presented 

b.  Proposed  approach  (or,  in  the  alternative,  the 
means  for  establishing  the  approach) 

c.  Present  or  potential  capability  to  accomplish  this 

Evaluation  team  members  may  use  information  obtained  from  the 
RFP  and  associated  documents,  the  bidders'  v/ritten  and  oral 
proposals  and  independent  or  prior  information.  However,  in 
the  latter  case,  any  significant  or  critical  information  must 
be  considered  by  all  members  of  the  factor  team.  Emphasis 
will  be  placed  on  the  sub-factor  being  rated,  but  the  impact 
of  other  factors  will  not  be  ignored.  Comparative  weighting 
of  individual  sub-factors  and  the  weight  of  each  factor  team 
members ' s views  will  be  within  the  responsibility  of  the 
factor  team. 

C.2.5  The  results  of  the  factor  team  evaluations  will  be 
summarized  and  reviewed. 

C.2.5.1  The  summary  process  will  include  the  necessary  combining 
of  the  individual  factor  ratings  to  achieve  an  overall  rating, 
with  consideration  of  the  interaction  of  factors,  their  indi- 
vidual significance  (acceptability) , and  the  "summation" 
according  to  previously  established  weights. 
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C.2.5.2  In  general,  the  range  .of  v;eights  among  the  factors 
will  be  no  more  than  2:1. 

C.2.5.3  The  reviev;  process  will  necessarily  provide  for 
latitude  in  reflecting  the  results  of  the  reviev;,  such  as 
reassessi.ng  of  v/eights  or  reexamination  of  specific  areas 
of  the  evaluation;  and,  such  changes  will  be  incorporated, 
v;ith  the  supporting  basis  in  the  record  of  the  evaluation. 

C.3  Comment  on  Factor  and  Sub-factor  Structure 

All  of  the  factors  in  the  evaluation  are  set  out  in  the 
Proposal  Evaluation  Factors  documents,  and  are  largely 
self-explanatory.  The  rationale  of  this  particular 
organization  of  the  sam.ple  is  based,  primarily,  on  the 
sub-factors,  the  factors  being  ad.ministrative  groupings 
for  convenience  in  organization  of  the  various  documents 
and  the  evaluation.  Generally,  t.he  factor/sub-f actor  organiza 
tion  is  based  on  significant  identifiable  effort/capability 
areas.  The  intent  is  to  select  comprehensive  exclusive/ 
inclusive  samples  of  the  total  program.  In  total,  the 
factor/sub-factors  are  inte.nded  to  include  samples  of  all 
significant  parts  of  the  program.  An  effort  has  been  miade 
to  minimize  duplication;  and,  v;ith  a few  intentional 
exceptions,  no  sub-factor  sam.ples  an  area  covered  by  another 
factor.  Similarly,  with  certain  intentional  exceptions, 
each  sub-factor  samples  on  the  basis  of  understanding, 
approach,  and  capability;  as  a result,  those  aspects,  v;hich 
include  "effective  experience,"  are  incorporated  into  specific 
substantive  areas  rather  than  amorphous  generalized  categories 
It  is  important  to  understand  that  the  distinction  am.cng  the 
factors  is  primarily  an  administrative  distinction  to  reflect 
the  differences  in  perspective  required  in  presentation  and 
evaluation . 
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D.  Use  of  this  Document 


D.l  General  Considerations 

D.1.1  As  noted  above,  the  Proposal  Evaluation  Factors  docu- 
ment covers  all  of  the  factors  to  be  considered  in  the 
evaluation  and  are  intended  to  provide  a substantially 
complete  framework  for  the  preparation  of  the  proposal. 
Because  the  background  and  perspective  against  v/hich  they 
were  prepared  is  set  out  in  this  part,  the  other  related 
documents,  and  the  briefings  for  the  bidders,  it  is 
extremely  important  that  this  interpolation  be  understood 
and  preserved  during  the  preparation  period  by  assuring  that 
substantive  contributors  have  sufficient  access  to  the 
overall  framework.  As  a general  guide,  cross  reference 
should  be  made  to  the  applicable  points  included  in  the 
document  on  Scope  of  Effort. 

0 

D.2  Content  Considerations 

D.2.1  Because  the  evaluation  is  organized  on  a factor/sub- 
factor basis,  it  is  important  that  the  treatment  of  each 
factor  be  substantially  complete  as  possible.  The  several 
factors  are  necessarily  interrelated  and  interacting,  but 
repetition  is  not  required;  hov/ever,  where  significant 
material  considered  necessary  to  assure  understanding  of 
a factor  is  contained  in  another  factor,  specific  (and 
annotated)  cross-references  should  be  furnished.  All  members 
of  the  evaluation  team  will  have  access. to  the  complete 
proposal ,'  and  it  is  expected  that  certain  factors  will  be 
reviewed  jointly. 

D.-2.2  Each  factor  and  sub-factor  is  defined  by  a specific 
descriptive  sentence  setting  out  the  standard  against 
which  each  bidder  will  be  measured.  Because  the  factor 
statements  reflect  the  fact  that  the  factor  is  primarily 
an  administrative  convenience,  no  specific  presentation 
with  regard  to  the  overall  factor  is  required.  Each  sub- 
factor, however,  will  be  the  subject  of  a specific  rating, 
and  the  proposal  content  should  reflect  the  necessity  for 
furnishing  the  evaluation  team  with  sufficient  information 
to  establish  a comparative  position  on  each  sub-factor 
THE  BASIC  ORGANIZATION  OF  THE  PROPOSAL  MUST  BE  ON  A SUB- 
FACTOR BASIS. 
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D.2.2.1  It  should  be  specifically  noted  that  the  sub-paragraphs 
under  each  sub-factor  are  provided  for  the  purpose  of  amplifying 
and  defining  the  standard,  that  is,  to  present  the  types  of 
questions  v/hich  need  to  be  answered  to  establish  the  rating  on 
the  sub-factor.  It  is  not  required  'to  ansv/er  each  and  all  of 
these  subsidiary  questions.  Nor  is  it  required  that  the 
presentation  be  limited  to  these  particular  subsidiary  questions. 
In  some  cases,  a subsidiary  question  v;ill  be  a critical  de- 
terminant; in  other  cases,  an  effective  presentation  may  be 
accomplished  by  partial  treatm.ent.  It  was  not  intended  tc 
specify  the  proposal  content  through  these  subsidiary  questions. 

D.2.2.2  These  subsidiary  questions  are  presented  in  three  for.ms . 
One  form  is  that  of  a question,  such  as,  "Do  you  consider  the 
proposed  schedule  realistic?  too  tight?  If  nor,  what  do  ycu 
propose  and  v;hy?"  The  second  general  form  is  that  of  a state- 
ment calling  for  comment  or  the  furnishing  of  certain  informa- 
tion, such  as,  "Testing  facilities  (floor  space,  equipment, 
personnel)."  The  third  general  form  is  that  of  a statem.ent 
commenting  on  a problem  area  or  outlining  a proposed  solution 
which  furnishes  a basis  for  conment,  such  as,  "The  Advisory 
Committee  may  raise  a technical  question  which  requires 
investigation  in  order  to  allov;  the  committee  to  m.ahe  a decision 
at  the  next  meeting.  The  contractor  v;ill  furnish  the  necessary 
personnel  to  develop  each  of  the  conflicting  points  of  view 
independently.  " 

D.2.2.3  As  noted  elsewhere,  the  relative  v;eighting,  or  signifi- 
cance, of  each  sub-factor  v/ithin  a factor  is  v/ithin  discretion 
of  the  factor  team  performing  that  part  of  the  evaluation.  In 
most  cases,  the  sub-factors  represent  approxim.ately  equivalent 
areas  of  significance.  The  amount  of  subsidiary  questions 
furnished  does  not  necessarily  reflect _ an  unusually  high  or 
low  v;eight. 

D.2.3  Latitude  is  encouraged  in  the  treatm.ent  of  the  content 
presented  v/ith  respect  to  the  sub-factors.  VThere  alternate 
assumptions  or  conclusions  to  clarify  ambiguities  or  fully 
establish  the  position  is  considered  advisable,  no  arbitrary 
limitation  is  im^posed.  The  substantive  objective  is  to  provide 
information  as  a basis  for  evaluation,  and  this  procedure 
is  intended  only  as  a minim.um  framework  to  increase  the 
efficiency  and  effectiveness  of  the  proposing  and  evaluating 
process . 

D.3  Physical  Considerations 

D.3.1  Primary  concern  v;ith  regard  to  the  physical  format  and 
organization  should  be  to  facilitate  the  work  of  the  evalua- 
tion team,  recognizing  the  circumstances  and  procedures  v;hich 
will  be  used. 
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D.3.2  It  is  suggested  that  the  proposal  use  the  same  major 
numerical  and  subject  headings  of  the  Proposal  Evaluation 
Factors  document  to  avoid  the  confusion  of  an  additional 
numbering  system.  This  should  be  extended  only  to  the 
factors  and  sub-factors,  and  no  requirement  to  follow  the 
numerical  order  of  the  subsidiary  questions  within  a sub- 
factor is  intended.  Where  it  is  considered  desirable  to 
include  additional  material  v/ithin  the  framework  of  Sections 
I through  VII,  an  expanded  numbering  sequence  should  be 
used,  i.e.,  to  include  material  after  I.l,  but  before  1.1.1, 
use  a new  number,  1.1.0,  with  subsidiary  numbers,  I. 1.0.1, 

I . 1. 0 . 2 , etc. 

D.3.3  To  facilitate  the  physical  handling  of  the  proposal, 
certain  physical  divisions  are  requested. 

D. 3.3.1  Physically  bulky  material,  such  as  extensive 
personnel  or  facility  brochures,  or  supporting  descriptive 
documents  furnished  in  amplification,  should  be  treated  as 
physically  separate  appendices,  v/ith  suitable  identification. 
In  this  regard,  where  material  which  has  been  prepared  in 
another  connection  is  furnished,  a suitable  cover  sheet 
noting  limitations  of  application  may  be  used  to  avoid 
extensive  revision  and  republication. 

D.3.4  Provision  should  be  made  for  sufficient  indexing  and 
cross-reference  summ.aries  to  facilitate  reference  to 
particular  parts  of  the  proposal.  Unless  considered 
desirable,  pagination  need  not  be  accomplished.  It  is 
recommended  that  covers  of  separate  documents  contain 
clear  reference  to  the  numerical  and  subject  heading. 


* 
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REVISIONS  TO  PROPOSAL  PREPARATION  PROCEDURES 


A.  Page  5.  Delete  C.2.2,  and  substitute  the  following: 

C.2.2  V/here  factor  team  members  require  specific 
additional  information  for  their  evaluation,  specific 
written  questions  will  be  directed  to  the  specific 
proposer  through  the  contracting  officer. 

B.  Page  6,  C.2.3,  lines  6 and  7.  Place  period  (.)  after 
"scales,”  and  delete  the  remaining  material. 

C.  Page  6,  C.2.4,  lines  24  to  27.  Delete  "Comparative 
weighting"  and  add  "Comparisons,"  and  delete  "weight" 
and  add  "efforts." 
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Note:  The  items  listed  belov;  provide  a catagorized  summary  of  the 

services  and  materials  required  on  this  program. 


Phase  One:  Planning  and  Design,  and  preliminary  evaluation. 

Item  1.  Plan,  organize,  staff,  direct,  and  control,  including 
. both  schedule  and  costs. 

Item  2.  Provide  reports,  liaison,  and  services  for  purposes 
of  review,  coordination,  and  approvals  and  accept- 
ances, including  the  following: 

a.  Periodic  and  special  formal  and  informal 
management  progress  reports,  final  report. 

b.  Periodic  and  final  scheduling  reports, 
including  projected  schedule  for  the  next  phase. 

c.  Periodic  and  final  cost  reports,  including 
projected  costs  for  the  next  phase. 

d.  Periodic  and  special  formal  and  inform.al 
technical  progress  reports  and  final  report, 
including  proposed  program  for  the  next  phase. 
(Note:  the  substantive  v;ork  to  be  reported 
here  is  to  be  performed  under  items  3 through 
7)  . 


Item  3.  Review  and  structure  the  proposed  objectives  in 
terms  of  the  experimental  setting  and  related 
background,  and  establish  an  overall  plan  and 
set  of  priorities. 

Item  4.  Review  and  structure  the  proposed  set  of  experiments, 
and  related  background  in  terms  of  the  proposed 
objectives,  and  provide  a reference  base. 


Item  5.  Develop  the  detailed  evaluation  requirem^ents  (problem 
definition  and  experimental  or  study  design) , in 
relation  to  collection  and  analysis. 

Item  6.  Develop  the  detailed  data  collection  process, 
including  instruments,  identification  of,  and 
access  to,  sources,  and  m.anaging  the  process; 
and  collect  the  data. 
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Item  7:  Develop  the  detailed  data  analysis  process; 

carry  out  the  analyses;  present  findings; 
present  recommendations. 
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Phase  Tv;o : Prototype  Evaluation  Process  Testing 

Item  8.  Same  as  Item  1. 

Item  9.  Same  as  Item  2,  but  change  reference  to  Items 
10  through  14. 

Item  10.  Refine  and  revise  the  work  accomplished  under 
Item  3,  as  related  to  the  further  objective  of 
designing  and  testing  a prototype  of  the  complete 
evaluation  process. 

Item  11.  Refine  and  revise  the  v;ork  accomplished  under 
Item  4,  as  related  to  the  further  objective  of 
preparing  a formal  framework  within  which  the 
objectives,  organizations,  and  programs  may  be 
related  to  the  evaluation  process. 

Item  12.  Refine  and  revise  the  work  accomplished  under 
Item  5,  as  related  to  the  further  objective 
to  achieve  a comprehensive,  detailed  design 
to  meet  both  basic  data  (base  line  data)  and 
special  data  requirements. 

Item  13.  Refine  and  revise  the  v;ork  accomplished  under 
Item  6,  as  related  to  the  further  objective 
of  collecting  both  basic  and  -special  data. 

Item  14.  Refine  and  revise  the  work  accomplished  under 
Item  7,  as  related  to  the  analysis  of  both 
basic  and  special  data;  and  provide  findings 
and  recommendations . 


Phase  Three;  On-line/Turnkey  Evaluation 
Items  15 

thru  21.  Revision  and  refinement  of  respective  prior 

items  in  relation  to  the  additional  objective 
of  an -on-line  turnkey  evaluation  process. 
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stateme:it  of  v;ork 

(with  illustrative  sub-items  for  Phase  One) 

Note:  The  items  listed  below  provide  a categorized 

sumjnary  of  the  services  and  materials  required 
on  this  program. 

Phase  One  Planning  and  Design,  and  Preliminary  Evaluation 

1.  Plan,  organize,  staff,  direct,  and  control,  including 
both  schedule  and  costs. 

a.  Provide  the  necessary  services  and  materials  for 
management  of  the  overall  program,  including  a 
management  plan. 

b.  Prepare  a preliminary  detailed  schedule,  and 
provide  progressive  analysis  and  rescheduling, 
including  proposed  schedule  for  Phase  Tv;o . 

c.  Prepare  periodic  cost  analyses  and  projections, 
including  proposed  costs  for  Phase  Tv/o . 
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Provide  reports,  liaison,  and  services  for  purposes 
of  review,  coordination,  and  approvals  and  accept- 
ances, including  the  following: 

Provide  services  and  materials  necessary  for  liaison 
with  ETIP,  PAA,  and  relevant  other  individuals  and 
organizations . 

a.  Periodic  and  special  formal  and  informal  manage- 
ment progress  reports,  final  report 


a(l)  . 

Submit  program  management  plan  at  or  before 
completion  of  3d  month,  and  report  modifica- 
tions as  required. 

a(2)  . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

b.  Periodic  and  final  scheduling  reports,  including 
projected  schedule  for  the  next  phase' 


b(l)  . 

Submit  projected  Phase  One  schedule  at  or 
before  completion  of  3rd  month,  and  report 
modifications  as  required. 

b(2). 

Submit  projected  Phase  Two  schedule  at  or 
before  completion  of  9th  month. 

b(3)  . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

c.  Periodic  and  final ‘cost  reports,  including  projected 
costs  for  the'  next  phase. 


c(l)  . 

Submit  projected  Phase  One  costs  at  or  before 
completion  of  3rd  month,  and  report  modifi- 
cations as  required. 

c(2)  . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

d.  Periodic  and  special  formal  and  informal  technical 
progress  reports,  final  report,  including  proposed 
program  for  the  next  phase.  (Note:  The  substantive 
work  to  be  reported  here  is  to  be  performed  under 
Items  3 through  7.) 
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Submit  reports,  including  formal  reports  for  the  items  in  the 
follov;ing  table  on  or  before  completion  of  the  month  (s) 
specified : 


3a 

3rd  and  6th 

6a 

6th 

b 

3rd 

b 

- 

c 

6th 

c 

6 th 

d 

9th 

d 

- 

4a 

3rd  and  6th 

7a 

6 th 

b 

6th  and  12th 

b 

12th 

5 a 

3rd  and  6th 

c 

12th 

b 

6th 

d 

15  th 

c 

6 th 

e 

15  th 

d 

12  th 
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Item  3.  Review  and  structure  the  proposed  objectives  in 
terms  of  the  experimental  setting  and  related 
background,  and  establish  an  overall  plan  and 
set  of  priorities. 

a.  Review,  through  docum.ents  and  interviev;s , the 
objectives  of  ETIP,  PAA,  and  other  relevant 
individuals  and  organizations,  including 
professional  standards  of  evaluation,  and  prepare 
a description  and  supporting  analysis  suitable 
for  providing  a basis  for  planning  and  setting 

of  priorities  for  Phase  One. 

b.  Based  on  the  above,  identify,  and  prepare  pre- 
liminary recommendations  and  supporting  detail 
requirements  for,  those  specific  evaluations 
which  should  be  accomplished  during  Phase  One, 
including  both  exploratory  (and/or  descriptive) 
evaluations  and  a priori  proposition  testing 
evaluations . 

c.  Based  on  the  above,  prepare  preliminary  statement 
of  requiremients  for  the  pilot  evaluation  system 
in  Phase  One. 

d.  Based  upon  the  above  and  the  other  activities 
in  Phase  One,  prepare  a preliminary  statem.ent 

of  the  requirements  for  the  prototype  evaluation 
system  in  Phase  Two,  together  with  supporting 
analysis. 
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Item  4.  Reviev;  and  structure  the  proposed  set  of  experiments, 
and  related  background  in  terms  of  the  proposed 
objectives,  and  provide  a reference  base. 

a.  Reviev;,  through  documents  and  interviews,  the 
specific  procurement  experiments  (past,  present, 
and  proposed) , and  the  related  programs  and 
organizations*,  and  prepare  a description  and 
supporting  analysis  suitable  for  planning 

the  activities  of  Phase  One  in  conjunction  v/ith 
the  work  in  Item  3a. 

* (AGENCY  IMPACT  ONLY)  V/i.th  prim^ary  emphasis 
on  PAA,  and  not  on  industry  sector. 

(COlblERCIAL  IMPACT  ONLY)  V7ith  primary  emphasis 
on  industry  sector,  and  not  on  PAA. 

b.  Prepare  a preliminary  framiework  m.odel  v/hich 
provides  a base  for  synthesizing  objectives, 
organization,  program.s , and  the  evaluation 
process  (system),  v;ith  supporting  analysis. 
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Develop  the  detailed  evaluation  requirements  (problem 
definition  and  experimental  or  study  design)  , in 
relation  to  collection  and  analysis. 

a.  Prepare  a preliminary  design  of  both  exploratory 
(and/or  descriptive)  and  a priori  evaluations  to 
be  conducted  during  Phase  One,  including  1)  state- 
ment of  problem  or  hypothesis,  2)  supporting 
theories,  and  3)  definitions  of  variables  and 
parameters . 

b.  Based  on  the  above,  prepare  a detailed  design  of 
each  evaluation,  with  special  emphasis  on  specific 
procurement  experiments,  including  1)  choice  of 
experimental  design,  2)  (where  appropriate) 
sam.pling  strategies,  and  3)  data  collection  and 
analysis  methods. 

c.  Prepare  a preliminary  design  of  the  pilot  evalua- 
tion system  in  Phase  One,  including  data  collection 
and  analysis. 

Based  on  the  above,  prepare  a preliminary  design  of 
the  prototype  evaluation  system  in  Phase  Two, 
including  both  basic  and  special  processes . 


d. 


Item  6 


V. 


386 

RFP  6-35756 


Develop  the  detailed  data  collection  process,  in- 
cluding instruments,  identification  of,  and  access 
to,  sources,  and  managing  the  process;  and  collect 
the  data. 

a.  Design  and  develop  the  data  collection  process 
required  for  the  specific  Phase  One  evaluations. 

b.  Based  upon  the  v;ork  in  Items  3,  4,  and  5,  collect 
additional  data  required  for  specific  exploratory 
and/or  descriptive)  evaluations. 

c.  Design  and  pilot  test  the  instruments  necessary  for 
specific  a priori  proposition  testing  evaluations. 

d.  collect  the  data  reauired  for  specific  a priori 
proposition  testing  evaluations. 
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Item  7.  Develop  the  detailed  data  analysis  process;  carry  out 

the  analyses;  present  findings;  present  recommendations. 

a.  Design  and  develop  the  data  analysis  process  re- 
quired for  the  specific  Phase  One  evaluations. 

b.  Carry  out  the  specific  Phase  One  exploratory  (and/or 
descriptive)  evaluations,  and  present  findings. 

c.  Carry  out  the  specific  Phase  One  a priori  propositioi 
testing  evaluations,  and  present  findings. 

d.  Analyse  the  activities  of  Phase  One  which  affect  the 
design  of  the  evaluation  system,  and  prepare 
recommendations . 

e.  Analyse  the  activities  of  Phase  One,  and  prepare 
recommendations  for  Phase  Tvzo . 


Phase  Tv;o : 
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Prototype  Evaluation  Process  Testing 


Item  8.  Same  as  Item  1. 


Item  9.  Same  as  Item  2,  but  change  reference  to  Items  10 
through  14 . 


Item  10. 


Refine  and  revise  the 
Item  3,  as  related  to 
designing  and  testing 
evaluation  process. 


work  accomplished  under 
the  further  objective  of 
a prototype  of  the  complete 


Item  11.  Refine  and  revise  the  v;ork  accomplished  under  Item  4, 
related  to  the  further  objective  of  preparing  a 
formal  framework  v;ithin  v;hich  the  objectives,  organi- 
zations, and  programs  may  be  related  to  the  evaluation 
process . 


Item  12.  Refine  and  revise  the  v/ork  accomplished  under  Item  5, 
as  related  to  the  further  objective  to  achieve  a 
comprehensive  detailed  design  to  meet  both  basic 
data  (base  line  data)  and  special  data  requirements. 


Item  13.  Refine  and  revise  the  v/ork  accom.plished  under  Item  6, 
as  related  to  the  further  objective  of  collecting  both 
basic  and  special  data. 


Refine  and  revise  the  v;ork  accomplished  under  Item  7, 
as  related  to  the  analysis  of  both  basic  and  special 
data;  and  provide  findings  and  recommendations. 


Item  14 
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Phase  Three:  On-Line/Turn-Key  Evaluation 


Items  15  through  21.  Revision  and  refinement  of  respective 

prior  items,  in  relation  to  the  addi- 
tional objective  of  an  on-line/turn- 
key  evaluation  process. 


$ 
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PROPOSED  SCHEDULE  - OVERALL 
{Agency  Impact  &.  Cofnmercial  Impact) 


ITEM  DESCRIPTION 

PHASE  ONE 

1.  PROGRAM  MANAGEMENT 

2.  REP  ORTS  & REVIEWS 

3.  OBJECTIVES 

4.  BACKGROUND 

5.  EVALUATION  DESIGN 

6.  DATA  COLLECTION 

7.  DATA  ANALYSIS 

PHASE  TWO 

8.  PROGRAM  MANAGEMENT 

9.  REPORTS  & REVIEWS 

10.  OBJECTIVES 

11.  BACKGROUND 

12.  EVALUATION  DESIGN 

13.  DATA  COLLECTION 

14.  DATA  ANALYSIS 
PHASE  THREE 

15.  PROGRAM  MANAGEMENT 

16.  REPORTS  & REVIEWS 

17.  OBJECTIVES 

18.  BACKGROUND 

19.  EVALUATION  DESIGN 

20.  DATA  COLLECTION 

21.  DATA  ANALYSIS 

NOTE:  (FORMAL)  PROGRESS  REPORT 

□ FINAL  REPORT 


ONLY 
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PROPOSED  SCHEDULE  - PHASE  ONE 


(Agency  impact  & Commercial  impact) 

Tasks,  plus 

selected  Months 

sub-tasks 


1.  PROGRAr'/l  MANAGEMENT 
Phase  Two  cost  proposal 

2.  REPORTS  & REVIEWS 
Design  of  reporting  system 

3.  OBJECTIVES 
Preliminary  analysis 

Identification  of  specific  Phase  One  evaluations 
Preliminary  evaluation  system  objectives 
Evaluation  system  objectives 


1 I 2 I 3 I 4 
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V 


V 


V 
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4.  BACKGROUND 

Preliminary  analysis 

Analysis  of  relevant  organizations 

Preliminary  framework 

5.  EVALUATION  DESIGN 

Analysis  of  Phase  One  evaluations 

Design  of  Phase  One  evaluations 
(including  sampling  strategies) 

Preliminary  design  of  evaluation  system 
(including  collection  & analysis) 

Design  of  evaluation  system 
(including  basic  & special) 


6.  DATA  COLLECTION 
General  data  collection 
Instrument  design  & pilot  test 
Data  on  specific  experiments 

7.  DATA  ANALYSIS 
Exploratory  analysis  & findings 

A priori  analysis  & findings 
Analysis  of  evaluation  system 
Recommendations 
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I . 


I.l 


1.2 


1.3 


1.4 


PROPOSAL  EVALUATION  FACTORS 


Management 

This  factor  is  a measure  of  the  background 

and  overall  management  capability  of  the  bidder. 

Evaluation  Management  Experience 

■^his  sub-factor  measures  the  bidder's  experience 
and/or  capability  of  the  personnel  he  v/ill 
use  on  thi.s  program. 

Importance  of  Program  to  the  Bidder 

This  sub-factor  is  a measore  of  the  degree  to 
which  the  proposal  represents  the  approval  and 
direction  of  the  company  rather  than  the  efforts 
of  professional  proposal  v/riters , and  also  is  a 
measure  of  the  warranty  offered  by  the  company 
to  back  the  program  with  the  necessary  resources. 

Program  Management 

This  sub-factor  is  the  m^easure  of  the  bidder's  own 
proposed  method  for  planning,  organizing,  and 
controlling  the  program. 

Schedule  and  Cost  Control 

This  factor  measures  the  bidder's  ability  to 
predict  the  time  and  cost  requirements  of  his 
efforts,  to  plan  the  interrelation  of  the  several 
parts,  and  to  exercise  the  necessary  control. 


I 
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II.  Reports  and  Reviev/s 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  government  is 
able  to  effectively  and  progressively  review 
a very  large  scale  effort  v/ith  a minimum  of 
people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  v/ith  those  of 
other  related  individuals  and  organizations. 

11. 1 Reports  and  Liaison  with  the  Experimental 
Technology  Incentives  Program  (ETIP) . 

This  sub-factor  measures  that  part  of  the 
overall  factor  v/hich  relates  to  the 
Experimental  Technology  Incentives  Program 
(ETIP) . 

11. 2 Reports  and  Liaison  with  the  Program  Administrative 
Agency  (PAA) 

This  sub-factor  measures  that  part  of  the  overall  facto 
which  relates  to  the  program  administrative  agency 
(and,  where  appropriate,  its  subelemients  or  related 
organizations) . 

II.  3 Liaison  v/ith  Others 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  individuals 
and  organizations  other  than  those  described 
above . 

II. 4 Approvals  and  Acceptances 

This  sub-factor  measures  the  bidder's  understanding 
and  acceptance  of  the  necessity  for  progressive 
and  substantial  approval  and  acceptance  during 
each  phase,  as  well  as  at  the  end  of  each  phase. 
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III.  1 


III.  2 


III . 3 


III.  4 


III. 5 


This  factor  is  a measure  of  the 
undcrstandinc;  of  the  objectives 
(and  related  programs) . 


bidder's  overall 
of  this  procurement 


ETIP  Evaluation  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  specific  objectives  of  this  procurement. 

Program  Administrative  Agency  (PAA.)  Objectives 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  specific  objectives  of  the 
program (s)  and/or  experiment (s)  that  are  to  be 
evaluated. 

Other  Relevant  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  relevant  and  related  objectives,  goals, 
criteria,  standards,  and  the  like,  including  the 
state  of  the  art  in  administrative  experimentation 
(and/or  evaluation) . 

I 

Phase  Tv;o  Objectives 


This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  incorporation  of  the  results  of 
Phase  One  in  the  restated  objectives  for  Phase  Tv/o . 

Phase  Three  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  reformulation  of  objectives  in 
Phase  Three. 
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IV. 


IV.  1 


IV.  2 


IV.  3 


IV.  4 


IV.  5 


Background  Information 

This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  background  and  characteristics 
of  this  procurement. 

ETIP 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  ETIP  and  the  specific  program (s)  and/or 
experiment (s) , in  terms  of  past  history,  present 
status  and  plans  for  the  future. 

Program  Administrative  Agency  (PAA) 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  program  administrative 
agency  (and,  where  appropriate,  its  subelem.ents 
or  related  organizations),  its  organization, 
policies  and  procedures. 

Other  Relevant  Background 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  programs,  organization,  policies 
and  procedures  of  other  relevant  organizations. 

Phase  Two  Background 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  prepare  a comprehensive  framework 
or  model  of  the  program(s)  and/or  experiment (s) 
and  the  evaluation  process. 

Phase  Three  Background 

This  sub-factor  is  a measure  of  the  bidder's  aware- 
ness of  possible  further  refinements  in  the  model 
necessary  for  the  continuation  of  the  evaluation 
process , 
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V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder’s  capabilities 
to  carry  out  the  evaluation  design  requirements. 

V.l  Exploratory  (Descriptive)  and  A Priori  Proposition 

Testing  Hypotheses,  Variables  and  Parameters 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  define  the  detailed  evaluation  objectives. 

V.2  Experimental  and/or  Study  Designs 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  define  the  overall  experimental 
and/or  study  design. 

V.3  Data  Collection  Design 

This  sub-factor  is  a measure  of  the  bidder ' s ability 
to  design  the  overall  data  collection  process , including 
sources  of  information,  sam.pling  strategies,  timing, 
etc . 

V.4  Data  Analysis  Design 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  design  the  overall  data  analysis  process, 
including  processing  and  analysis,  and  presenting 
results  and  recommendations . 

V.5  Phase  Two  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  redefine  the  detailed  experimental  and/or  study 
design  to  reflect  the  results  of  Phase  One,  the 
several  objectives,  and  model  of  the  process  in  a 
comprehensive  detailed  design  to  m.eet  both  basic 
data  (base  line  data)  and  special  data  requiremients . 

V.6  Phase  Three  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to 
further  redefine  the  above,  to  provide  for  a detailed 
design,  and  necessary  supporting  materials,  v/hich  v7ould 
allow  an  on-going  evaluation  process. 
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VI.  Data  Collection  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
acquisition  of  data  on  the  variables  and  parameters 
of  interest. 

VI. 1 Instrument  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  develop  (and  test)  the  various  questionnaires , 
protocols,  schedules,  and  the  like  required  for  data 
collection. 

VI. 2 Information  Sources 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  identify  the  various  inform.ation 
sources,  and  to  gain  access. 

VI. 3 Data  Collection 

This  sub-factor  is  measure  of  the  bidder's  capability 
to  organize,  staff,  train,  direct  and  control  the 
personnel  who  obtain  the  data  by  survey,  interview, 
observation,  or  use  of  records. 

VI. 4 Phase  Two  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  requirements  of  the  Phase  Tv;o 
collection  of  both  basic  (base  line)  and  special 
data. 

VI. 5 Phase  Three  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  plan  against  the  requirement  to  design  an 
ongoing  collection  process. 
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Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
analysis  of  the  dfita  and  to  present  the  results  and 
recommendations . 


Analysis 

This  sub-factor  is  the  measure  of  the  bidder's  ability 
to  identify  and  apply  the  appropriate  analytical 
(including  statistical)  techniques  to  the  data 
collected . 

Exploratory  (and/or  Descriptive)  Findings 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  exploratory 
findings . 

A Priori  Proposition  Testing  Findings. 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  a priori 
findings . 

Recomm.endations 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  relationship  betv/een  the 
results  of  Phase  One  arid  the  work  in  Phase  Two, 
in  terms  of  additional,  revised,  or  deleted 
hypotheses,  changes  in  objectives  or  background 
information,  and  the  relative  feasibility  and/or 
usefulness  of  alternative  designs  for  evaluation. 
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2/1/76 

PROPOSAL  EVALUATIOn  FACTORS 
(V7ith  Illustrative  Questions) 


Note:  These  factors  and  sub-factors  apply  to  both  agency 

and  commercial  impact  proposals.  Where  specific 
illustrative  questions  apply  primarily  to  either  agency 
or  commercial  impact  only,  they  are  so  identified. 

I . Management 

This  factor  is  a measure  of  the  background 
and  overall  management  capability  of  the  bidder. 

I.l  Evaluation  Management  Experience 

This  sub-factor  measures  the  bidder's  experience 
and/or  capability  of  the  personnel  he  will 
use  on  this  program. 


1.1.1  Discuss,  with  specific  examples,  recent  experience  in  the 
management  of  field  experiments,  projects,  and/or  programs. 
Indicate  size  and  complexity  of  program,  extent  of  ov;n  responsi- 
bility for  design  of  the  program  (or  experiment)  and  the  evalua- 
tion. Indicate  by  tabulation  of  cross-references  other  sub- 
sections of  this  proposal  v/here  you  have  referred  to  any  program 
discussed  here.  Outline  the  original  contracted-for  schedules 
and  costs  and  requirements,  and  explain  changes.  If  you  have 
had  any  "bad"  experience  here,  discuss  the  reasons  and,  if 
appropriate,  indicate  how  you  would  prevent  them  from  occurring 
on  this  program.  If  appropriate,  you  should  provide  the 
names,  addresses,  and  telephone  numbers  of  cognizant  COTR's. 

1.1.2  If  you  do  not  have  the  kind  of  experience  discussed  above, 
discuss  your  basis  for  believing  that  you  can  m.eet  the  require- 
ments of  this  program.  Discuss  in  terms  of  the  personnel  you 
intend  to  use. 

1.1.3  How  V7ill  you  combine  and  coordinate  the  efforts  of 
staff  members  who  are  familiar  wU-h  the  procurement  process, 
the  program  administrative  ager»cy/  or  the  commercial  sector 
with  those  staff  members  v;ho  hn^'c  the  technical  skills  in 
evaluation? 
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This  sub-factor  is  a measure  of  the  degree  to 
which  the  proposal  represents  the  approval  and 
direction  of  the  company  rather  than  the  efforts 
of  professional  proposal  writers,  and  also  is  a 
measure  of  the  v/arranty  offered  by  the  company 
to  back  the  program  with  the  necessary  resources. 


1.2.1  What  is  the  relative  size  of  this  program  with  respect 
to  the  overall  sales  of  the  company?  If  your  com.pany  has 
significant  decentralization  or  there  are  other  factors  v/hich 
should  be  presented  for  clarity,  please  furnish  this  back- 
ground. If  the  proportionate  size  is  relatively  large,  why 
do  you  think  that  the  problems  associated  with  the  "single 
product"  will  not  arise?  If,  on  the  other  hand,  the  propor- 
tionate part  is  relatively  sm.all,  v;hat  assurance  is  there 
that  this  program  v/ill  not  be  downrated  by  others  after  the 
contract  is  signed? 

1.2.2  Discuss,  if  appropriate,  hov/  this  program  fits  in  with 
the  long  range  planning  of  your  com.pany.  What  is  its  signifi- 
cance to  your  future  position?  If  you  envisage  an  increase  in 
programs  of  this  type,  what  assurance  is  there  that  key  personnel 
on  this  program,  will  not  be  v/ithdrawn  for  nev;er  programs? 

1.2.3  It  should  be  noted  that  the  extent  and  nature  of  your 
effort  in  proposing  v/ill  be  considered  in  the  evaluation  of 
this  sub-factor. 
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1.3  Program  Management 

This  sub-factor  is  the  measure  of  the  bidder's  ovm 
proposed  method  for  planning,  organizing,  and 
controlling  the  program. 


1.3.1  It  is  proposed  that  the  planning  and  carrying  out 

of  the  preparation  of  this  proposal  provides  an  opportunity 
for  the  bidder  to  demonstrate  his  capability  in  planning  a 
complex  effort.  Please  describe  briefly  your  process  and, 
particularly,  the  extent  to  v/hich  it  is  a "pilot"  of  the 
process  you  will  use  in  managing  this  program. 

1.3.2  Detail  your  proposed  organization  by  time  phases, 
including  buildup,  division  of  responsibility;  generally 
describe  its  operation,  how  it  is  organized,  and  special 
features  not  self-explanatory.  Can  the  effectiveness  of 
your  proposed  organization  be  projected  from  its  performance 
in  putting  together  this  proposal?  If  the  organization  you 
propose  is  essentially  "new,"  either  in  form  or  in  the 
persons  assigned,  comment  on  the  extent  to  which  you  have 
been  able  to  "test"  it  out  in  preparing  this  proposal. 
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1.4  Schedule  and  Cost  Control 

This  factor  measures  the  bidder's  ability  to 
predict  the  time  and  cost  requirements  of  his 
efforts,  to  plan  the  interrelation  of  the  several 
parts,  and  to  exercise  the  necessary  control. 

1.4.1  A detailed,  thorough  schedule  provides  not  only  a 
comprehensive  plan  but  also  a pov;erful  method  of  controlling 
and  reviev/ing  the  progress  of  the  program.  The  proposed 
schedule  does  not  include  a detailed  breakdown  of  all  of 
the  interrelated  areas  but  does  furnish  a broad  guide  for 
planning  purposes.  In  general,  do  you  think  the  schedule 

is  realistic?  Is  it  too  tight?  V7hat  additional  assumptions 
are  required?  What  conditions  or  events  v/ill  affect  the 
schedule? 

1.4.2  Discuss  the  method  of  preparation  of  the  schedule  you 
V7ill  use,  how  you  v;ill  update  it,  how  you  v;ill  use  it. 

1.4.3  Prepare  a supporting  analysis  and  sumimary  of  any  areas 
which  you  believe  reflect  a significant  relative  cost  advantage 
or  disadvantage  on  your  part.  Indicate  to  v;hau  extent  your 
in-house  or  subcontracting  plans  will  take  advantage  of 
in-being  capabilities  and  at  v;hat  potential  savings.  To 

what  extent  have  provisions  for  contingencies  been  m.ade? 

1.4.4  Discuss  the  form  and  timing  of  your  detailed  cost 
proposal  for  the  major  field  data  gathering  activity  in 
the  second  and  third  phases. 

1.4.5  While  the  overall  schedule  reflects  the  requirem.ent 
tq  design  an  on-going  evaluation  system,  there  is  an 
important  requirement  to  provide  during  Phase  One  (and  each 
subsequent  phase)  specific  outputs  in  the  form  of  evaluations 
of  selected  specific  experiments.  How  will  you  assure  this? 

1.4.6  The  timing  of  the  experiments  varies  across  the  several 
PAA’s.  What  advantages  or  disadvantages  do  you  see  in  this? 

1.4.7  It  is  anticipated  that  another  ("agency  i.mpact") 
contractor  will  be  primarily  responsible  for  evaluating  the 
intervention  and  proximal  imipacts  of  the  several  experim.ents  ; 
it  is  also  anticipated  that  some,  if  not  all,  ccmiaercial 
impacts  will  substantially  lag  the  experiments.  What  advantages 
or  disadvantages  do  you  see  in  this?  (COMMERCIAL  IMPACT  OMLY) 
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II.  Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  government  is 
able  to  effectively  and  progressively  review 
a very  large  scale  effort  v/ith  a minimum  of 
people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  with  those  of 
other  related  individuals  and  organizations. 

II. 1 Reports  and  Liaison  with  the  Experimental 

Technology  Incentives  Program  (ETIP) 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  the 
Experimental  Technology  Incentives  Program 
(ETIP)  . 


11. 1.1  Discuss  generally  the  extent  and  nature  of  liaison 
with  both  the  Contracting  Officer  (CO)  and  the  Contracting 
Officer's  Technical  Representative  (COTR)  that  you  envision 
in  this  program.  Discuss  generally  the  information  needs  of 
both  government  and  contractor  personnel;  who  will  establish 
what  the  needs  are  and  how  they  will  be  satisfied? 

11. 1.2  It  would  be  useful  if  the  reporting  system  were 
designed  in  a way  that  facilitated  an  ongoing  process 
evaluation  of  a)  the  experiments  conducted  by  the  program 
administrative  agency,  and  b)  the  evaluation  effort  of  the 
contractor  himself.  The  data  provided  by  such  a system  could 
serve  as  the  basis  for  a descriptive  case  history  of  the 
experiments  and  their  evaluation.  Discuss  generally  how 
feasible  this  proposal  is,  and  how  it  might  be  accomplished. 
You  may  wish  to  relate  it  to  your  discussion  in  IV. 1.2,  and 
comparable  discussions. 
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II . 2 


11. 2.1 

both  management  personnel  and  technical  or  specialized  staff. 

How  will  you  reconcile  the  desirability  of  direct:,  informal  com- 
munication with  the  necessity,  in  at  least  some  cases,  for 
observing  formial  clearance  and  approval  "through  channels"? 

11. 2. 2 The  evaluation  effort  may  sometimes  be  perceived  as 
threatening  by  various  personnel  in  the  program  administra- 
tive agency.  Discuss  hov/  the  reporting  and  liaison  arrange- 
ments can  be  developed  to  reduce  the  likelihood  of  this. 

11. 2. 3 Discuss  the  differences,  if  any,  in  how  you  will  relate 
to  the  several  PAA.  To  what  extent  v;ill  these  be  separately 
identifiable  efforts? 

II.  2. 4 It  is  anticipated  that  another  ("agency  im.pact") 
contractor  will  be  prim.arily  responsible  for  coordination 
with  the  several  PAA.  Vlhat  do  you  think  of  this,  a.nd  how 
will  you  deal  with  it?  (C02-D1ERCIAL  IMPACT  ONLY) 


Reports  and  Liaison  with  the  Program  Administrative 
Agency  (PAA) 

This  sub-factor  measures  that  part  of  the  overall  facto 
which  relates  to  the  prcgra::.  adm.inistrative  agency 
(and,  v/here  appropriate,  its  subelem.ents  or  re  laced 
organizations) . 

Discuss  generally  the  extent  and  nature  of  liaison  v/ith 
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II. 3 Liaison  with  Others 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  individuals 
and  organizations  other  than  those  described 
above . 

11. 3.1  It  is  anticipated  that  there  will  be  related  programs 
and  program  evaluations  being  carried  out  by  the  governmient 
agencies  either  in-house  or  through  the  use  of  an  outside 
consultant  or  contractor.  Discuss  the  nature  and  form  of 
the  relationship  you  see  with  them.  How  will  you  ensure 
that  the  necessary  coordination,  avoidance  of  duplication, 
overlap,  or  gaps,  etc.,  is  accomplished? 

11. 3.2  To  some  extent,  it  is  expected  that  individual  companies 
(e.g.,  manufacturers  of  products  which  are  the  subject  of  pro- 
gram experiments),  industry  associations,  and  others  miay  desire 
or  require  a liaison  relationship.  Comment. 

11. 3. 3 Reporting  and  liaison  relationships  may  arise  with 
individuals  and  organizations  with  professional  interests 
(e.g.,  professional  societies,  conferences,  professional 
journals,  consumer  or  public  interest  groups)  in  the  pro- 
gram or  program  evaluation.  Discuss. 


II.  3. 4 It  is  anticipated  that  another  ("commercial  impact") 
contractor  will  be  primarily  responsible  for  coordination  with 
individual  companies,  etc.  Comment.  (AGENCY  IMPACT  ONLY) 

II. 3.5  It  is  anticipated  that  the  bidder  will  be  primarily 
responsible  for  coordination  with  individual  com.panies , etc. 
Comment.  (COMMERCIAL  IMPACT  ONLY) 
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II. 4 Approvals  and  Acceptances 

This  sub-factor  measures  the  bidder's  understanding 
and  acceptance  of  the  necessity  for  progressive 
and  substantial  approval  and  acceptance  during 
each  phase,  as  v/ell  as  at  the  end  of  each  phase. 

II. 4.1  It  is  intended  that  the  formal  approval  and  acceptance 
of  the  v;ork  of  the  contractor  for  each  phase  v;ill  be  based, 
primarily,  upon  the  several  reports  submitted  during  and 
at  the  end  of  the  phase,  v/ith  final  approval  at  the  end  of 
the  phase.  It  is  intended,  however,  that  this  final  approval 
and  acceptance  will  be  largely  a formal  confirmation  of 
already  established  acceptable  performiance  and/or  results. 
This  is  required  not  only  to  overcome  the  substantial  delays 
of  tandem  or  sequential  review,  but  also  to  avoid  re’.;ork, 
redesign  and  argument  under  pressure  of  time  and  after  the 
fact.  You  should  discuss  your  understanding  of  this,  the 
advantages  and  disadvantages,  and  your  willingness  to  accept 
the  contractual  and  schedule  implications,  that  is,  that  this 
means  that  the  delivery  dates  are  dates  of  delivery  of 
approved  items,  not  merely  dates  for  submission  of  items 
with  an  open  time  term. 


49 


RFP  6-35756 


^07 


« 

N 

« 

m 

m 

a 

m 

a 

a 

a 

a 

a 

a 

a 

a 

a 

a 


III . 


Objectives 


This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  objectives  of  this  procurement 
(and  related  programs). 


111.1  ETIP  Evaluation  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  specific  objectives  of  this  procurem.ent . 

111.1. 1 The  specific  objectives  for  this  procurement,  and 
related  objectives,  including  the  overall  objectives  of 
ETIP,  appear  in  various  forms  in  the  several  parts  of  this 
bid  set,  and  in  other  sources,  and  reflect  a variety  of 
authors,  circumstances  and  purposes.  It  is  proposed  that 
it  is  neither  necessary  nor  desirable  for  ETIP  to  prepare 

a single,  integrated,  fully  rationalized,  and  "authoritative" 
statement  of  the  objectives  for  the  guidance  of  the  bidders 
(or  the  successful  bidder) . Comment  on  why  you  believe  this 
is  (or  is  not)  justified? 


III. 1.2  It  is  expected  that  the  various  stated  objectives 
may  not  appear  (or  be)  consistent;  how  will  you  handle  this? 
For  example,  a high  credibility  (a  priori  proposition  testing) 
evaluation  of  a specific  experiment  is  desired  but  it:  is  not 
possible  to  achieve  this  (v;ith  reasonable  time  and  money)  . 

For  example,  a complete  and  candid  description  of  how  a 
particular  experiment  was  carried  out  may  reflect  gross 
incompetence  or  dereliction  on  the  part  of  specific  ETIP 
(or  program  administrative  agency)  personnel.  For  example, 
the  results  of  a particular  evaluation  may  disclose  a 
systematic  defect  in  the  past  performance  of  the  agency 
(or  a contractor) . 


111.1. 3 It  is  anticipated  that,  for  a particular  experiment, 
there  may  be  as  many  as  four  parallel  objectives:  a)  to 
describe  what  happened,  what  went  well,  what  went  wrong; 

b)  to  determine  whether  the  result  was  "caused"  by  the 
experimental  manipulation;  c)  to  demonstrate  the  ability 
to  "evaluate"  scientifically  with  credible  results;  d)  to 
claim  credit  for  the  experiment  and/or  the  results.  How 
does  one  determine  the  relative  importance  of  these  several 
objectives  for  a specific  experiment  over  the  set  of  all 
comparable  experiments? 

111. 1.4  It  is  anticipated  that  ETIP  evaluation  objectives  or 
priorities  may  shift  during  the  course  of  the  evaluation 
effort.  How  does  one  ensure  that  he  will  be  sensitive  to 
such  shifts  and  how  does  one  deal  with  such  shifts?  How  would 
you  go  about  improving  the  clarity  of  ETIP's  evaluation 
objectives? 
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III.  2 Program  Administrative  Agency  (PAA)  Objectives 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  specific  objectives  of  the 
program (s)  and/or  experiment (s ) that  are  to  be 
evaluated . 

III.  2.1  Questions  comparable  to  those  suggested  under  III.l 
may  appear  here.  Comment.  What  special  proble.ms,  if  any, 
do  you  anticipate  because  of  the  multiple  PAA? 

III.  2. 2 How  will  you  reconcile  conflicts  between  ETI?  a.nd 
the  program  administrative  agency  objectives  for  a particu- 
lar experiment?  PAA  objectives  for  the  experiment  and 
other  PAA  objectives?  What  kinds  of  problems  will  com.e 
up?  How  do  you  plan  to  deal  v;ith  them? 


III.  2. 3 It  is  anticipated  that,  during  the  period  that  the 
work  on  this  contract  is  accomplished,  there  may  be 
changes  in  policy  or  procedures  or  of  key  personnel  of  the 
program  administrative  agency.  How  v/ill  you  deal  v:ich  this? 
For  exam.ple,  the  "results"  of  a particular  experim:enu  may  be 
sufficiently  "obvious"  to  the  agency  to  no  longer  m.ake 
necessary  any  further  evaluation.  Will  a continuation  of 
the  evaluation  be  undertaken  under  any  circum.stances?  And, 
if  so,  how  v/ould  you  justify  it  and/or  obtain  agency  co- 
operation? 


III. 2. 4 Suppose  the  agency  adds  new  objectives  or  objectives 
not  presently  contemplated  are  uncovered  v/hich  represent 
potentially  significant  changes  in  the  direction  or  scope 
of  the  present  evaluation  program?  Hov;  will  you  react  to 
this? 


III.  2. 5 It  is  anticipated  that  another  ("agency  im.pact") 
contractor  will  be  primarily  responsible  for  exam.ining  the 
objectives  of  the  PAA.  Comment.'  (C0>'E-1ERCIAL  IMPACT  OMLY) 


51 


409 

RFP  6-35756 


m 

n 


\ 


m 

« 

a 

a 

a 

a 

a 

a 

a 

a 


III. 3 Other  Relevant  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  relevant  and  related  objectives,  goals, 
criteria,  standards,  and  the  like,  including  the 
state  of  the  art  in  administrative  experimentation 
(and/or  evaluation) . 

111. 3.1  It  is  proposed  that  the  objectives  of  ETIP  and  PAA 
do  not  exist  in  isolation,  that  both  specific  and  general 
objectives  of  a wide  variety  of  other  individuals,  organisa- 
tions, and  larger  cominunities  will  necessarily  interact  with 
the  present  program.  For  example,  personal  and  career  object- 
ives of  key  individuals.  For  example,  political,  social',  and 
legal  objectives  of  the  federal  governm.ent . For  example, 
objectives  expressed  or  implied  in  the  state-of-the-art  and 
professional  standards  in  experimental  and  evaluation  research. 
Of  what  significance  are  these,  and  how  do  you  expect  to  deal 
with  them? 

111. 3.2  Your  understanding  of  the  state-of-the-art, 
particularly  in  administrative  experimentation  and/or 
evaluation,  v;ill  be  measured  by  your  discussion  of  relevant 
questions  raised  elsewhere.  If  you  consider  it  appropriate, 
you  may  discuss  this  here,  and  you  may  wish  to  cross-reference 
significant  material  which  appear^  elsewhere. 

111. 3. 3 Are  there  relevant  professional  standards  in 
experimentation  that  you  must  observe  that  may  conflict 
with  ETIP  or  PAA  objectives?  How  will  you  handle  this? 

You  may  wish  to  relate  your  answer  to  II. 3. 

111. 3. 4 It  is  anticipated  that  another  ("commercial  impact") 
contractor  will  be  primarily  responsible  for  exam.ining  the 
objectives  of  individual  companies  and  others  in  the 
commercial  sector.  Comment.  (AGENCY  IMPACT  ONLY) 

111. 3.5  It  is  anticipated  that  the  objectives  of  individual 
companies,  associations,  etc.,  may  significantly  affect  not 
only  the  experiments  and  their  evaluation  but  also  the 
activities  of  the  bidder.  Comment.  (COMMERCIAL  IMPACT  ONLY) 
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III. 4 Phase  Two  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 

to  anticipate  the  incorporation  of  the  results  of 

Phase  One  in  the  restated  objectives  for  Phase  Two. 

III.  4.1  It  is  proposed  that,  as  a result  of  the  work  in 
Phase  One,  a reasonable  description  and  integration  of  the 
several  objectives  will  have  been  realized,  and  that  this 
will  be  used  as  a basis  for  the  several  other  tasks  but 
also  as  a basis  for  refinement,  revision,  and  additions 
during  Phase  Tv/o . In  addition  to  this  continuing  interest 
in  the  objectives  introduced  in  Phase  One,  it  is  proposed 
that  a separate  objective  of  Phase  Two  is  to  design  and 
test  a prototype  of  the  complete  evaluation  process 
necessary  to  meet  the  several  objectives.  It  is  expected 
that  this  may  include  at  least  tv/o  separately  identifiable 
parts,  as  follows:  a basic  evaluation  process  to  include 

those  evaluation  activities  ^vhich  deal  with  key,  predictable, 
identifiable,  continuing,  common  objectives  for  which  the 
process  can  be  "institutionalized";  and  a special  evalua- 
tion process  to  include  those  evaluation  activities  which 
do  not  rrieet  the  above  requirem.ents  because  of  specialized, 
one-time,  or  changing  objectives.  If  you  believe  your 
discussion  elsewhere  does  not  adequately  reflect  your  under- 
standing of  this,  you  may  comment  here. 

III. 4. 2 What  activities  during  Phase  One  v/ill  be  expected 
to  contribute  to  the  refinement  of  Phase  Tv;o  objectives? 

What  other  sources  may  be  expected  to  provide  input?  You 
may  wish  to  cross-reference  your  comments  under  IV. 4,  or 
elsewhere . 
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III. 5 Phase  Three  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  reformulation  of  objectives  in 
Phase  Three. 

III. 5.1  In  addition  to  a continuation  of  the  concern  with  the 
objectives  of  the  previous  two  phases,  it  is  proposed  that 
the  objective  of  Phase  Three  will  be  to  provide  a firm  basis 
for  any  further  related  evaluation  efforts,  including  any  of 
the  following:  a)  a standby  or  "now  to"  capability  to  design 

and  carry  out  evaluations  for  comparable  future  experiments; 
b)  an  on-line,  turnkey  data  gathering  and  analysis  process 
which  might  be  implemented  by  the  PAA;  c)  an  assessment  of 
the  need  (or  lack  of  need)  for  other  specific  or  general 
future  evaluation  efforts.  How  realistic  do  you  think  this 
is,  and  to  v/hat  extent  will  the  prior  phases  prepare  you  to 
successfully  define  and  then  meet  these  objectives? 


$ 
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Background  Information 

This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  background  and  characteristics 
of  this  procurement. 

ETIP 

This  sub-factor  is  a measure  of  the  bidder ' s under- 
standing of  ETIP  and  the  specific  progra.m(s)  and/or 
experiment (s ) , in  terms  of  past  history,  present 
status  and  plans  for  the  future. 

IV. 1.1  Analogous  to  the  comments  in  III. 1.1,  it  is  not 
proposed  to  provide  a complete  and  comprehensive  descriptio.n 
of  ETIP  and  the  specific  program(s)  and/or  experimient  ( s ) for 
the  guidance  of  the  bidders  (or  the  successful  bidder)  . 

Discuss  the  adequacy  of  your  present  understanding  of  ETIP 
and  the  specific  programs;  if  you  believe  it  is  not  adequate, 
how  do  you  propose  to  correct  this  consistent  v.'ith  the 
requirements  of  Phase  One? 

IV.  1.2  It  is  assumed  that  a reasonably  com.plete  and  current 
description  of  at  least  the  specific  program.(s)  a.nd/or 
experiment (s ) will  be  desirable,  if  not  necessary,  during 
Phase  One  (and,  perhaps,  on  a continuing  basis)  for  use 
not  only  by  the  bidder  but  also  by  ETIP.  How  can  this  be 
accomplished?  What  forms  and  procedures  do  you  visualize 
will  be  used  to  provide  for  an  up-to-date  record?  How 
will  inputs  (revisions)  be  made?  How  will  outputs  be 
available? 

IV. 1.3  The  availability  of  such  an  information  base  (and 
system)  would  necessarily  increase  the  opporounities  to 
make  corrections  in  ongoing  activities  (experim.ents ) 
and  may  suggest  other  improvements  and  chances  in  the 
program.  To  what  extent  do  you  see  it  as  your  responsibility 
to  facilitate  this  process?  What  effect  v/culd  these  additicr.a 
changes  have  on  your  evaluation  work? 

IV. 1.4  Discuss  your  understanding  of  the  overall  set  of 
programs  and/or  experiments  to  be  eyaluated,  including  at 
least  the  following:  the  set  of  individual  product  procure- 
ments under  life  cycle  costing  (LCC)  both  planned  and 
implemented;  the  FSS  decision  to  adopt  LCC  as  a procurement 
practice;  the  several  workshops  and  the  training  program  as 
planned  and  implemented  in  all  FSS  offices  around  the  country, 
other  planned  or  implem.ented  procurement  experiments,  such 
as  value  incentive  contracting.  Can  these  be  described  or 
modeled  in  such  a form  as  to  disclose  their  interactions, 
or,  conversely,  can  specific  experim^ents  or  programs  be 
defined  in  such  a way  as  to  allov;  separate  treatment? 


IV. 
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IV. 2 Program  Administrative  Agency  (PAA) 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  program  administrative 
agency  (and,  where  appropriate,  its  subelements 
or  related  organizations),  its  organization, 
policies  and  procedures. 

IV. 2.1  Questions  comparable  to  those  suggested  under  IV. 1.1 
may  appear  here.  Comment.  What  special  problems,  if  any, 
do  you  anticipate  because  of  the  multiple  PAA? 

IV. 2. 2 As  a further  restatement  of  the  above,  if  you  do  not 
have  a present  acquaintance  v;ith  the  program  administrative 
agency  (and  its  constituent  elements)  , hov;  v;ill  you  acquire 
the  necessary  background? 

IV. 2. 3 At  least  some  parts  of  the  organization,  policies, 
and  procedures  of  the  program  administrative  agency  will 
be  critically  related  to  the  present  program.  For  example, 
those  elements  directly  involved  in  the  experiments,  and 
procurement  policies  and  procedure.  Discuss  your  under- 
standing of  these  areas.  If  you  do  not  have  prior  experience, 
v;hat  effect  will  this  have  on  your  capability? 

• « 

IV. 2. 4 It  is  anticipated  that  another  ("agency  impact") 
contractor  will  be  primarily  responsible  for  obtaining 
information  about  and  access  to  the  PAA.  How  will  you  deal 
with  this?  (COMI-IERCIAL  IMPACT  ONLY) 
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IV. 3 Other  Relevant  Background 

This  sub-factor  is  a measure  of  the  bidder's  unde 
standing  of  the  programs,  organization , policies 
and  procedures  of  other  relevant  organize tions . 


IV. 3.1  It  is  anticipated  that  information  about  and  access 
to  organizations  other  than  ETIP  and  P.'Ji  may  be  importa.nt 
to  this  program.  For  example:  other  governm.ent  age.ncies 
which  are  the  end  users  of  the  products  procured;  other 
government  agencies  v/ith  legal,  financial,  policy,  or 
other  responsibilities  which  interact  v/ith  the  program 
area;  specific  com.panies,  industry  segments,  a.nd  industry 
associations;  consum>er  organizations,  safety,  advertising, 
news  media,  and  the  like;  other  ETIP  or  program,  or 

evaluation  contractors.  Discuss  your  understanding  of  the 
potential  effects  of  these  interactions  and  hov/  you  v/ill 
deal  with  them. 

IV.  3.2  It  is  anticipated  that  another  ("commercial  imicact") 
contractor  will  be  primarily  responsible  for  obtaining  info 
about  and  access  to  individual  com.panies  and  others  in  the 
commercial  sector.  Comm.ent.  (AGSb'CY  III? ACT  OITLY) 

IV.  3. 3 It  is  anticipated  that  the  bidder  v/ill  be  primarily 
responsible  for  obtaining  inform.ation  about  and  access  to 
individual  companies  a.nd  others  in  the  comm.ercial  sector. 
Comment.  (COmERCIAL  IMPACT  ONLY) 
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IV. 4 Phase  Two  Background 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  prepare  a comprehensive  framework 
or  model  of  the  prograra(s)  and/or  experiment  (s ) 
and  the  evaluation  process. 


IV. 4.1  It  is  anticipated  that,  after  Phase  One,  the  bidder 
v/ill  be  capable  of  preparing  a formal  framework  v;ithin 
which  the  objectives,  organizations,  and  programs  may  be 
related  to  the  evaluation  process.  Such  a framework  would 
not  only  integrate  the  background  from  Phase  One  but  provide 
a convenient  and  efficient  capability  for  revision,  additions 
and  modifications  without  having  to  start  over  again.  Ccrrmen 
on  the  feasibility  of  this;  if  you  can,  suggest  hcv;  this 
might  be  done  or  how  one  v/ould  determine  whether  it  was 
successful  or  not?  You  may  wish  to  cross-reference  your 
comments  on  III. 4. 

IV. 4. 2 Will  the  distinction  between  agency  impact  and 
commercial  impact  require  significantly  different 
frameworks?  If  tv;o  separate  frameworks  are  developed, 
to  what  extent  will  it  be  necessary  and/or  feasible  and/or 
desirable  to  relate  the  frameworks? 

IV. 4. 3 Discuss  the  same  question  with  respect  to  the 
distinction  among  the  several  PAA. 
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IV. 5 Phase  Three  Background 

This  sub-factor  is  a ir.easure  of  the  bidder's  av;are- 
ness  of  possible  further  refinements  in  the  model 
necessary  for  the  continuation  of  the  evaluation 
process . 

IV. 5.1  It  is  proposed  that  the  model,  and  its  supporting 
descriptive  raaterial  would  provide  a sufficient  basis  for 
a turnkey  operation  so  that  PAA  personnel  could,  with 
modest  specialized  assistance,  carry  out  on-going 
evaluations  for  comparable  programinatic  extensions. 

Comment  on  this. 

IV.  5. 2 It  is  anticipated  that  the  specific  P.AA  v;culd  be 
primarily  interested  in,  and,  be-,  logically,  the 
appropriate  organization  to  m^anage , the  on-going  "agency 
impact"  evaluation  system  (or  that  part  concerned  with 
the  specific  agency).  Comment.  (AGENCY  IMPA.CT  OELY) 

IV. 5. 3 It  is  anticipated  that  significant  parts  of  the 
"commercial  impact"  evaluation  system  may  already  be  withi.n 
the  present  capability  (or  future  plans)  of  related  o 
tions  (e.g..  Department  of  Commerce,  industry  associa 
etc.).  VThat  advantages  or  disadvantages  do  you  see  in  this 
possibility,  and  how  will  you  react  to  it?  (CC2-D1ZRCIAL 
IMPACT  ONLY) 
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V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder's  capabilities 
to  carry  out  the  evaluation  design  requirements. 

V. l  Exploratory  (Descriptive)  and  A Priori  Proposition 

Testing  Hypotheses,  Variables  and  Parameters 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  define  the  detailed  evaluation  objectives. 

VI.  1.1  It  is  proposed  that  the  various  evaluation  design  require- 
ments v;ill  include  problems  ranging  over  several  dimensions, 

and  combinations  of  dimensions,  including  the  following: 
a)  from  exploratory  and/or  descriptive  evaluations  (e.g., 
how  did  a specific  experiment  get  started;  what  happened; 
what  are  the  more  significant  secondary  effects)  to  a priori 
proposition  testing  (e.g.,  did  specific  intervention  X "cause" 
the  observed  result  Y) ; b)  from  very  specific,  identifiable 
experiments  (manipulations  and/or  effects)  to  general,  diffuse, 
overall  changes;  c)  from  post  hoc  evaluations  of  completed 
experiments  through  evaluations  of  ongoing  experiments,  to 
planning  and  design  of  future  experiments.  Commient  on  your 
understanding  of  this. 

V.l. 2 At  least  initially,  the  bidder  will  be  called  upon  to 
accomplish  a preliminary  systems  design  or  study.  If  your 
evaluation  experience  has  been  primarily  in  response  to 
detailed  evaluation  requirements  provided  in  the  RFP  or 
RFQ,  discuss  the  relevance  of  your  experience  or  capability 
to  the  requirements  of  this  program. 

V.’1.3  Discuss  your  approach  to  the  detailed  definition  of 
a research  (or  experimental  or  evaluation)  problem;  comment 
on  theory  building,  definition  of  variables  and  param.eters , 
etc.  If  considered  necessary,  distinguish  the  approach  to 
a priori  proposition  testing  from  that  used-  in  exploratory 
and/or  descriptive  evaluation. 
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This  sub-factor  is  a measure  of  the  bidder's 
ability  to  define  the  overall  experimental 
and/or  study  design. 


V.2.1  Discuss  the  problem  (s)  of  and  approaches  to  experir.ental 
and/or  study  design.  Alternatively,  ycu  nay  v;ish  to  dem.onstra 
your  capability  by  reference  to  the  releva.nt  education  and 
experience  of  your  staff.  It  is  suggested  that  you  may  v;ish 
to  distinguish  among  the  several  kinds  of  desig.ns  v;hich  may 
be  appropriate  for  meeting  the  several  kinds  of  objectives 
v/hich  are  the  outcome  of  the  process  discussed  in  V.l. 

V.2.2  Discuss  the  problem  (s)  of  and  approaches  to  ide.ntifying 
and  controlling  for  the  potential  effects  of  parameters. 

V.2.3  it  is  proposed  that  the  bidder  can  design  an  overall 
evaluation  system  and,  in  parallel,  carry  cut  evaluations  of 
specific  selected  experiments  during  Phase  One.  Discuss 
the  advantages  and  disadvantages  of  this.  Ca.n  you  identify 
specific  experiments  which  can  be  evaluated  duri.ng  Phase 
One?  Can  you  identify  base-line  dara  v;hich  can,  or  should, 
be  collected  during  Phase  One?  v;hich  can,  or  should,  be 
collected  during  later  phases? 

V.2.4  To  what  extent  do  you  see  differences  among  the  several 
PAA  which  V7ill  affect  the  design  of  uhe  evaluation  system.? 

Will  differences  in  the  progress  in  initiating  experim.entis 
among  the  several  PAA.  be  an  advantage  or  disadvantage? 

V.2.5  It  is  anticipated  that  some  part  or  all  of  the  bidder's 
activity  in  this  area  will  be  dependent  upon  the  activities 
of  another  ("agency  impact")  contractor.  Hov;  critical  is 
this  from  a time  point  of  view?  from  an  interface  point  of 
view?  Will  segmentation  (by  industry,  technology,  market) 
provide  advantages  or  disadvantages?  to  what  extent  will 
longitudinal  designs  be  important,  and  v;ill  this  be  an 
advantage  or  disadvantage  (COilMERCIAL  IMPACT  ONLY) 


61 


419 

RFP  G-3575G 


I 


Dtal 

'Strata 

sli 

y 


of 


I 


ral 


I 

I 

I' 


I 

I 

I 

I 

I 

I 


V.3  Data  Collection  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  design  the  overall  data  collection  process,  including 
sources  of  information,  sampling  strategies,  timing, 
etc . 

V.3.1  Discuss  the  various  kinds  of  sources  or  m^ethods  of 
collecting  information  (e.g.,  observations,  questionnaires, 
interviews,  records)  in  terms  of  your  expectations  for 
their  use  here,  and  their  advantages  and  disadvantages. 

V.3.2  The  nature  and  extent  of  the  information  required  will, 
at  least  in  some  cases,  require  the  development  of  sampling 
strategies.  Describe,  by  reference  to  relevant  experience, 
if  appropriate,  your  capability  to  develop  sampling  strategies 
for  a priori  proposition  testing  evaluations.  You  may  wish 
to  discuss  the  approaches  used  for  exploratory  evaluation. 

You  may  wish  to  comment  on  some  of  the  sampling  strategies 
provided  as  illustrations  in  the  several  reference  documents. 

V.3. 3 Discuss  any  special  considerations  v;hich  may  be  intro- 
duced by  the  differences  among  the  several  PAA. 

V.3.4  It  is  anticipated  that  the  evaluation  of  agency  impact 
and  commercial  impact  v/ill  require  the  obtaining  of,  in- 
some  cases,  not  only  the  same  data  but  also  separate 
additional  data  from  coimnon  sources.  How  can  this  be 
coordinated  to  minimize  duplication  and,  particularly, 
the  burdening  of  sources? 
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V.4  Data  Analysis  Design 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  design  the  overall  data  analysis  process, 
including  processing  and  analysis,  and  presenting 
results  and  recommendations. 

V.4.1  It  is  proposed  that  this  part  of  the  process  should  be 
developed  concurrently  v/ith  those  discussed  in  V.2  and  V.3 
instead  of  waiting  until  the  data  is  collected.  Comment  on 
the  desirability,  necessity,  and/or  feasibility  of  this. 

V.4. 2 Specific  individual  experiments  and  specific  exploratory 
studies  may  have  not  only  different,  but  also  severable 
analytical  require.ments  (and  discussion  of  these  may  more 
appropriately  appear  in  VII).  It  is  expected,  however,  than 
there  may  be  some  degree  of  overlap  or  interrelation  among 
the  several  experiments  and  studies.  To  v:hat  extent  v;ould 
an  overview  of  the  overall  process  improve  the  ability  to 
perform  the  individual  analyses? 

V.4.3  It  is  proposed  that  the  quantity  and  variety  of  the 
various  data  collected  (and  to  be  collected  in  the  future) 
present  an  opportunity  for  careful  planning  with  respect 
to  formatting  identification,  quality  control,  standardization , 
etc.  To  what  extent  do  you  think  this  is  necessary  and/or 
feasible?  To  v;hat  extent  can  com.patability  be  achieved  wizr. 
related  existing  data  banks? 
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V.5  Phase  Two  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  redefine  the  detailed  experimental  and/or  study 
design  to  reflect  the  results  of  Phase  One,  the 
several  objectives,  and  model  of  the  process  in  a 
comprehensive  detailed  design  to  meet  both  basic 
data  (base  line  data)  and  special  data  requirements . 

V.5.1  Discuss,  or,  if  appropriate,  cross-reference  discussions 
elsewhere  of,  your  understanding  of  the  problems  outlined  in  V,1 
through  V.4  as  applicable  to  Phase  Two,  and  with  reference 
to  the  requirements  of  Phase  Two. 

V.5. 2 It  is  anticipated  that  the  bidder  v;ill  provide  a 
comprehensive  and  detailed  design  of  an  evaluation  system 
which  will  provide  a capability  for  both  basic  and  special 
evaluation.  Discuss  your  present  understanding  of  the 
form  and/or  extent  of  the  evaluation  system.  Discuss  the 
advantages  and  disadvantages  of  prototype  testing  the  system 
by  evaluating  specific  procurement  or  other  experiments? 
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V.6  Phase  Three  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to 
further  redefine  the  above,  to  provide  for  a detailed 
design,  and  necessary  supporting  materials,  v;!;ich  v;ouj.d 
allov;  an  on-going  evaluation  process. 

V.6.1  Discuss  or,  if  appropriate,  cross-reference  discussions 
elsewhere  v/hich  deraonstrate  your  ability. 

V.6. 2 Discuss  your  understanding  of  what  v/ill  be  necessary  to 
meet  turnkey  requirements.  V/hat  consideratio.n  of  agency  policies 
and  procedures  v;ill  be  required?  Hov;  v/ill  the  necessary  skills 
and  experience  be  transferred?  Hov;  will  previously  collected 
data  be  transferred?  What  coxuversion  of  instruments  anc.  sof~v/are 
will  be  required?  What  v/ill  be  the  effect  of  on-going  evaluauicns 
of  specific  experiments  on  the  transition  process?  and  vice  versa? 
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VI.  Data. Collection  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
acquisition  of  data  on  the  variables  and  parameters 
of  interest. 

VI . 1 Instrument  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  develop  (and  test)  the  various  questionnaires, 
protocols,  schedules,  and  the  like  required  for  data 
collection , 

VI . 1 . 1 Discuss  (you  may  use  an  example  and/or  reference  to 
prior  experience)  the  problems  and  solutions  which  you  con- 
sider critical  (and/or  characteristic)  in  instrumentation 
for  an  a priori  proposition  testing  type  of  evaluation. 

VI . 2 Discuss  the  differences,  if  any,  in  the  approach  to 
descriptive  or  exploratory  evaluation. 

VI. 1.3  Under  what  circumstances,  and  for  what  purposes, 
will  you  conduct  pilot  tests?  Validation  tests? 

VI. 1.4  It  is  anticipated  that  some  variables  of  interest 
will  be  of  ongoing  importance  (as  key  variables  or  parameters 
common  to  a number  of  experiments,  or  as  key  exploratory 
or  descriptive  variables)  in  a sense  comparable  to  "social 
indicators"  and  will  become  part  of  what  will  be  known  as 
"basic  data"  in  future  phases.  To  V7hat  extent  can  this  be 
anticipated,  and  what  difference  will  this  make  during  this 
phase? 

VI. 1.5  Discuss  your  understanding  of  the  special  requirements, 
if  any,  which  a government  agency  (or  its  contractor)  must 
meet  before  use  of  instruments  for  data  collection. 

VI. 1.6  What  advantages  or  disadvantages  are  introduced  by 
relevant  or  related  existing  data  banks  in  terms  of  the 
instruments  they  use  (e.g.,  various  definitions  of  industry 
classifications,  kinds  of  technology,  etc.)? 
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VI. 2 Information  Sources' 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  identify  the  various  information 
sources,  and  to  gain  access. 

VI. 2.1  Discuss  the  important  or  critical  problems  you  anticipate 
in  this  area  and  your  capability  to  deal  v/ith  them. 

VI. 2. 2 Discuss  the  various  strategies  for  identifying  sources 
of  information.  Distinguish,  if  appropriate,  betv/een  sou 
of  information  needed  for  evaluating  a specific  experimen 
with  those  to  be  used  for  a broad  descriptive  (case)  study. 

VI. 2. 3 Discuss  the  issues  of  confidentiality  and  privacy  v/ith 
respect  to  information  obtained  from  individuals.  Discuss 
the  issues  related  to  information,  in  general,  with  restrictions 
on  dissemination,  e.g.,  proprietary  inf orm.at io.n , certain 
types  of  personnel  and  financial  information. 

VI.  2. 4 Discuss  the  issues  involved  in  access  to  i.ndividuals 
and  to  records.  A special  issue  of  concern  is  the  demands 
on  the  time  of  individuals  in  the  program  ad.ministrat ive 
agency,  and  others  who  may  have  a considerably  less  direcn 
interest  in  the  evaluation. 


VI . 2 . 5 Much  background  and  parametric  inf  orm.ation  may  be 
available  in  some  form,  in  existing  records  v;hich  have  been 
collected  for  some  other  purpose;  how  will  you  identify  these 
and  what  advantages  or  disadvantages  do  they  present? 
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VI . 3 Data  Collection 

This  sub-factor  is  measure  of  the  bidder's  capability 
to  organize,  staff,  train,  direct  and  control  the 
personnel  v/ho  obtain  the  data  by  survey,  interview, 
observation,  or  use  of  records. 

VI. 3,1  It  is  anticipated  that  the  several  requirements  for 
data  collection  may  require  different  skills,  different 
organization,  and  different  methods  of  supervision  and 
control.  Discuss  by  example  or  reference  to  prior  experience 
your  capability  in  this  regard. 

VI . 3 . 2 With  reference  to  the  question  asked  in  VI. 1.4, 
v/hat  difference  in  approach  V70uld  be  v/arranted  with  respect 
to  "basic  data"? 

VI . 3 . 3 Discuss  or  describe  the  process  for  handling  data 
collection  (e.g.,  flowcharting).  Discuss  problems  of 
monitoring  and  evaluating  data  quality,  of  privacy  and 
confidentiality,  of  machine  acceptable  versus  non-machine 
acceptable  data  (e.g.  , unstructured  interviev/s)  . 
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VI.  4 Phase  Tv/o  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  requirements  of  the  Phase  Tv/o 
collection  of  both  basic  (base  line)  and  special 
data . 

VI . 4 . 1 It  is  proposed  that  the  bidder  v/ill  be  able  to 
institutionalize  the  collection  of  basic  data  (i.e.,  hey 
variables  and  parameters  common  to  a n'umber  of  present 
or  expected  experim.ents , or  key  exploratory  or  descriptive 
variables)  during  this  phase;  from  this,  base  line  data 
can  be  obtained  and  a "test"  of  the  process  can  be 
accomplished.  Comment  on  this. 

VI.  4. 2 It  is  proposed  that  there  v/ill  remain  other  require- 
ments not  included  within  the  above  (e.g.,  specific  one-tim.e 
requirements,  unusual  or  specialized  requirem.ents ; added  or 
revised  requirem.ents)  v/hich  cannot  or  should  not  be  com- 
mingled with  the  above.  Discuss  this. 
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VI . 5 Phase  Three  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  plan  against  the  requirement  to  design  an 
ongoing  collection  process. 

VI. 5.1  At  least  the  basic  data  collection  process  should 
be  sufficiently  stabilized  and  described  to  allov/  an 
ongoing,  turnkey  implementation.  Discuss  the  problems 
and  prospects  of  this. 

VI . 5 . 2 To  v/hat  extent  can  new  or  other  specialized  require- 
ments be  anticipated? 

VI . 5 . 3 Discuss  any  special  problems  with  respect  to  access, 
privacy,  confidentiality,  etc.,  which  may  be  introduced  when 
an  agency  takes  over  management  and  operation  of  the  evalua- 
tion system. 
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VII.  Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
analysis  of  the  data  and  to  present  the  results  and 
recommendations . 


VII. 1 Analysis 

This  sub-factor  is  the  measure  of  the  bidder ' s ability 
to  identify  and  apply  the  appropriate  analytical 
(including  statistical)  techniques  to  the  data 
collected . 


VII.  1.1  Discuss,  by  exam.ple  or  reference  to  prior  experience, 
your  ability  to  accomplish  the  various  analytical  tasks 
involved . 

VII. 1.2  For  at  least  some  experiments  (and/or  policy  changes), 
a significant  if  not  prim.ary  objective  is  econcm;ic.  Discuss 
your  capability  in  analysing  "cost/benefits,"  including 
problems  introduced  by  estimating,  allocating,  and  fore- 
casting costs  and  benefits,  both  direct  and  indirect. 

VII. 1.3  For  at  least  some  experiments  (and/or  policy  changes), 
a significant  if  not  primary  objective  is  to  bring  about 
some  change  in  the  policies  and  procedures  of  either  the 
program  administrative  agency  or  of  segm.ents  of  the  industry. 
Discuss  your  capability  in  analysing  administrative 
(behavioral)  changes. 

VII. 1.4  For  at  least  some  experim.ents  (and/or  policy  changes), 
a significant  if  not  primary  objective  is  to  bring  about  some 
change  in  the  use  of  technology  - change  in  the  rate,  appli- 
cation, etc.  Discuss  your  capability  an  analysing  technolo- 
gical change. 
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Exploratory  (and/or  Descriptive) 'Findings 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  exploratory 
findings . 


VII.  2.1  It  is  anticipated  that  in  many  areas  of  the  evalu- 
ation, it  will  be  neither  required  nor  feasible  to  develop 
findings  which  meet  the  more  stringent  formal  requirements 
of  a priori  proposition  testing.  To  what  extent  do  you 
consider  it  your  responsibility  to  reexamine  cases  where 
there  is  no  requirement,  especially  v;here  it  appears  that 
a priori  proposition  testing  ^ feasible? 

VII.  2. 2 You  may  wish  to  comment  on  some  of  the  requirements 
as  presently  described  in  the  several  reference  documents. 

VII.  2. 3 You  may  v/ish  to  discuss  the  various  forms  of 
presentation  (i.e.,  case  studies,  informal  reports, 
status  reports,  systems  studies,  etc.) 

VII. 2. 4 In  addition  to  exploratory  (and/or  descriptive)  findings 
with  respect  to  experiments  (and  their  related  parameters  and 
background) , it  is  proposed  that  the  evaluation  system  design 
is,  itself,  an  exploratory  or  descriptive  "finding."  What 
differences,  if  any,  will  be  required  in  the  presentation  and 
related  supporting  documentation  or  justification  of  the 
evaluation  system  design? 
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VII. 3 A Priori  Proposition  Testing  Findings. 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  a priori 
findings. 

VII.  3.1  It  is  anticipated  that  not  only  specific  exper i.m.ents 
(at  least  in  terras  of  their  pror.iinal  effects  or  impacts)  but 
also  other  specific  questions  v/hich  m.ay  be  subject 
to  study  or  experiment  will  require  (and  it  v/ill  be  feasi 
to  accomplish)  a priori  proposition  testing  evaluation, 
what  extent  do  you  consider  it  your  responsibility  to  re- 
examine cases  v/here  there  is  no  requiremen t?  Cases  v/here 
there  is  a requirement,  but  it  is  not  feasible? 

VII.  3. 2 You  may  wish  to  comme.nt  on  so.me  of  the  requirements 
as  presently  described  in  the  several  reference  docu.me.nts. 

VII.  3. 3 You  may  wish  to  discuss  the  various  fo2rms  of 
presentation . 

VII. 3. 4 Do  you  think  it  is  desirable  and/or  feasible  to  develop 
a procedure  or  process  v;hich  can  be  used  to  make  prelim.ina ry 
assessments  of  the  likelihood  that  a particular  proposed 
experiment  v/ill  be  capable  of  a priori  proposition  testing? 
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VII. 4 Recommendations 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  relationship  betv/een  the 
results  of  Phase  One  and  the  v;ork  in  Phase  Tv;o , 
in  terms  of  additional,  revised,  or  deleted 
hypotheses,  changes  in  objectives  or  background 
information,  and  the  relative  feasibility  and/or 
usefulness  of  alternative  designs  for  evaluation. 

# 

VII. 4.1  In  addition  to  the  specific  findings  discussed  in 
VII.  2 and  VII.  3,  it  is  anticipated  that  the  v;ork  in  each 
phase  will  provide  a basis  for  the  work  in  subsequent 
phases.  It  is  anticipated  that  this  input  to  the  next' 
phase  may  become  available  before  the  completion  of  all 
of  the  work  in  the  current  phase.  Comment  on  the  feasibility 
of  an  overlap,  the  advantages  and  disadvantages. 

VII. 4. 2 It  is  anticipated  that  a "final  report"  will  be 
prepared  at,  or  after,  the  completion  of  each  phase  v/hich 
V7ill  describe  v;hat  the  bidder  sets  out  to  do,  what  he  did, 
including  relevant  activities  of  others,  and  the  results 
(VII. 2,  VII. 3,  VII. 4).  Comment  on  the  relation  of  the 
final  report  to  the  several  other  forms  (and  timings)  of 
outputs . 
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NUMERICAL  V/EIGHTS  FOR  PROPOSAL  EVALUATION  FACTORS 


Factors 

Management 

This  factor  is  a measure  of  the  background 
and  overall  management  capability  of  the 
bidder . 

Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  Government  is 
able  to  effectively  and  progressively  reviev; 
a very  large  scale  effort  with  a minirr.um  of 
people,  and  of  the  bidder ' s ability  to 
coordinate  his  activities  v;ith  ti'icse  of 
other  related  individuals  and  organizations. 


Weights 


15 


Ob j ectives 


This  factor  is  a m.easure  of  the  bidder's  ove: 
all  understanding  of  the  objectives  of  this 
procurement  (and  related  programs) . 


20 


Background  Information 


This  factor  is  a m.easure  of  the  bidder's  ove: 
all  understanding  of  the  background  and 
characteristics  of  this  procurement. 


15 


Evaluation  Design 

This  factor  is  a m.easure  of  the  bidder's 
capabilities  to  carry  out  the  evaluarion 
design  requirements.  15 

Data  Collection  Process 


This  factor  is  a m.easure  of  the  bidder's  capa- 
bility to  carry  out  ail  of  the  functions 
necessary  to  the  acquisition  of  data  on  the 
variables  and  param.eters  of  interest.  10 

Data  Analysis  Process 

This  factor  is  a m.easure  of  the  bidder's  capa- 
bility to  carry  out  all  of  the  functions 
necessary  to  the  analysis  of  the  data  and  to 
present  the  results  and  recomm.endaticns . ‘ 10 


Seven  Evaluation  Factors  Total  100 
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SCOPE  OF  EFFORT 
Introduction 


A.  Purpose 


The  purpose  of  this  document  is  to  provide  a guide  to  the 
bidder  in  the  preparation  of  supporting  data  on  the  scope 
of  effort  ha  proposes. 

B.  Basis  for  Preoaration  of  This  Docum.ent 


This  document  is  intended  to  be  used  in  conjunction  vjith  the 
associated  Proposal  Preparation  Procedure,  Proposal  Evaluation 
Factors,  Schedule,  and  Statement  of  Work.  The  objective  is  to 
assist  the  bidder  in  presenting  his  supporting  data  on  scope 
of  effort  in  an  orderly  and  usable  form  so  that  it  may  be 
evaluated  concurrently  v;ith  his  written  (and  oral)  proposal. 

B.l  In  order  to  achieve  an  orderly  and  usable  form,  the  sup- 
porting data  on  scope  of  effort  must  be,  at  least  substantially, 
related  to  the  factor  (and  sub-factor)  categories  which  are  the 
basis  for  the  evaluation.  The  detailed  outline  furnished  in 
the  next . section  is  for  this  purpose. 

B.2  In  addition  to  the  above,  it  is  necessary  to  assure  that 
the  data  furnished  by  the  several  bidders  be  on  a reasonably 
comparable  basis.  To  accomplish  this  the  detailed  guidance 
included  in  this  Introductory  Section  is  furnished.  It  should 
be  understood  that  the  guidance  in  this  respect  is  necessarily 
arbitrary  in  the  interests  of  meeting  the  specific  objective. 

B.3  The  format  requirements  reflect  the  fact  that  evaluation 
team  members  need,  and  can  use,  only  selected  data,  primarily 
in  terras  of  man/months  of  effort  in  certain  areas  and  certain 
materials  costs.  A summary  analysis  of  the  scope  of  effort 
against  cost  data  will  be  accomplished. 
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To  assist  the  bidder  in  understanding  mtore  fully  the  part 
which  these  data  will  have  in  the  evaluation./  these  general 
comments  are  offered. 

C.l  This  type  of  program  which  establishes  a broad  objective 
but  provides  for  planning  and  design  to  work  out  v;ith  more 
detail  and  precision  the  mteans  for  acccmplish.ment  cannot  be 
costed  with  the  closeness  normally  experienced  on  procure.r.ents 
of  items  with  a stabilized  design.  For  this  reason  the 
bidders’  respective  total  cost  esti.mates  will  not  necessarily 
reflect  the  actual  realized  costs  to  the  govern.ment,  nor  do 
they  necessarily  reflect  the  relative  costs  of  the  bidders 
in  providing  a ccm.parable  capability.  Therefore,  relative 
total  cost,  as  such,  cannot  and  v/ill  not  be  given  any  direct 
weight  in  the  evaluation.  Cost,  in  terms  of  scope  of  effort, 
however,  will  aid  in  establishing  the  kind  and  extent  of  efforc, 
and  this  information  will  serve  to  further  defi.ne  the  "v/ord- 
pictures"  included  in  the  written  (and  oral)  proposals.  The 
formal  evaluation  of  relative  cost  and  associaced  areas  is 
included  as  part  of  o.ne  of  the  major  factors  in  the  evaluation 
(1.4),  and  further  reference  should  be  directed  there. 

C.2  T,vhile  the  major  impact  of  these  data  in  the  evaluation  v;ili 
be  in  providing  additional  perspective  to  the  v/ritten  (a.nd 
oral)  proposals,  certain  corollary  information  will  be  used. 

C.2,1  A measure  of  how  realistically  and  thoroughly  the  costing 
has  been  done  will  be  established.  A too  high  estimate  i.ndi- 
cates  the  obvious  disadvantages  of  "gold  plating"  or  a 
potentially  loose  operation.  In  contrast,  however,  a too  lev; 
estimate  presents,  in  the  long  run,  even  m.ore  serious  dis- 
advantages. First,  it  may  indicate  a serious  misundersta.ndir.g 
of  the  scope  and  objective  of  the  program.  Second,  it  may 
raise  a serious  question  as  to  the  bidder's  ability  to  plan 
and  estimate  costs,  thus  im>pairing  his  ability,  and  uhau  of 
the  government,  to  review,  predict  and  control  costs.  Finally, 
it  may  indicate  something  less  than  com^plete  candor  in  presentin 
the  cost  part  of  the  proposal. 
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C.2.2  The  data  presented  here  will  be  used  to  supplement  the 
data  furnished  v;ith  respect  to  cost  as  a factor  in  the  over- 
all evaluation. 

C.2.3  The  data  presented  here  will  be  used  in  negotiation  with 
the  successful  bidder  both  as  a basis  for  further  refining  the 
contractual  statement  of  work  and  in  establishing  the  contract 
price . 
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D.l  It  is  of  critical  importance  that  all  persons  directly  or 
indirectly  preparing  data  on  scope  of  effort  be  thoroughly 
familiar  with  the  rules  on  use  of  the  format.  This  is 
especially  true  because  the  figures  to  be  included  do  not, 
in  all  cases,  follow  the  accounting  distinction  between  direct 
and  indirect  costs.  Further,  certain  types  of  effort  are  to 
be  tabulated  as  separate  ite.ms.  An  effort  has  been  made  to 
use  format  and  terminology  as  close  to  the  general  practice 
as  possible.  tNlhiere  this  has  not  been  donfe , the  basis  has 
been  to  assume  that  useful  data  for  the  evaluation  can  be 
obtained.  Certain  distinctions,  such  as  direct  and  indirect 
costs,  have  little  significance  for  this  purpose.  Si.milarly, 
certain  areas  of  effort  can  only  be  evaluated  if  separated 
from  proportionately  large  associated  areas  of  effort. 

D.2  The  next  section  specifies  the  scope  of  effort  data  on 
a factor  by  factor  basis.  To  facilitate  the  evaluation,  chese 
data  should  be  collated  v;ith  the  respective  factors.  Because 
the  procedure  for  evaluation  provides  for  each  faccor  to  be 
evaluated,  substantially,  without  mandatory  recourse  to  other 
parts  of  the  proposal,  it  is  critically  i.mportanc  that  the 
data  on  scope  of  effort  and  the  v/ritten  proposal  cn  a specific 
factor,  taken  together,  is  ccmpleue  and  se  If -expla.natory . Care 
must  be  exercised  to  assure  that  the  evaluation  team  does  not 
inadvertently  penalize  the  v/ritten  proposal  because  of  an 
inadequate’ scope  of  effort. 

D.2.1  Under  each  factor,  the  scope  of  effort  should  be  set 
out  separately  for  the  first  phase  of  the  program.  V.'here 
the  bidder  proposes  to  begin  or  com.plete  items  in  another 
phase,  note  should  be  made  to  avoid  misunderstanding . 

D.3  It  is  expected  that  in  m.any  instances  the  scope  of  effort 
proposed  will  be,  literally,  the  bidder’s  present  estimace 
of  the  approximate  level  (or  scope)  of  effort  he  expects 
will  be  required  to  accomplish  what  he  proposes  to  do  in 
his  written  (and  oral)  proposal.  A certain  lauiuude  is 
expected,  and  these  data  on  scope  of  effort:  v;ill  be  used, 
as  stated  before,  primarily  to  assist  the  evaluation  team 
in  understanding  what  the  bidder  is  actually  proposing. 

There  are,  however,  certain  areas  which  may  prevent,  a clear 
understanding,  and  some  of  these  are  tabulated  below. 
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D.3.1  The  breakdown  in  the  next  section  is  a minimum  guide 
in  tabulating  the  scope  of  effort.  VThere  a particular  figure 
is  considered  unusually  small  or  large,  or  for  some  other 
reason  may  be  misleading,  it  should  be  footnoted  with  an 
explanation  or  further  breakdown. 

D.3.2  In  some  cases  identification  of  particular  types  of 
effort  with  a proposed  "associate"  or  subcontractor  v/ill  be 
of  assistance.  This  will  be  particularly  true  where  part  or 
all  of  an  effort  is  proposed  to  be  accomplished  by  some 
specialist  group. 

D.3.3  The  analysis  of  comparative  cost  advantage  will  not  be 
accomplished,  as  a separate  factor,  by  the  overall  evaluation 
team.  It  is  necessary,  however,  where  a bidder  is  estimating 
costs  which  are,  on  a relative  basis,  low  that  the  basis  for 
this  be  included  in  the  scope  of  effort  data  to  assure  that 
the  evaluation  team  understands  that  the  bidder  does  intend 
to  accomplish  what  he  has  said  in  his  written  proposal. 

D.3.4  Conversely,  where  a possible  misunderstanding  may  occur 
due  to  a lack  of  appreciation  by  the  evaluating  team  of  the 
cost  associated  with  accomplishing  a particular  part  of  the 
proposal,  the  bidder  should  make  this  clear. 

D.3.5  Man/month  estimates  should  be  divided  according  to 
distinct  classes  to  avoid  confusion  or  ambiguity  in  two 
respects.  First,  the  bidder's  data  should  distinguish  between 
direct  and  indirect.  Second,  the  data  should  distinguish 
between  senior  professional  personnel  and  supporting  technicians, 
draftsmen,  etc.,  and  between  various . management  personnel  and 
clerical  assistance,  etc. 

D.3.6  Any  further  analysis  considered  necessary  to  assure  that 
the  evaluation  team  properly  assesses  the  relative  scope  of 
effort  as  related  to  the  written  proposal  is  encouraged.  This 
may  be  particularly  true  in  terms  of  level  of  training  of 
personnel,  availability  of  facilities,  ability  to  control  design, 
and  the  like. 
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E . Summary  Data 

E.l  The  relationship  of  the  individual  scope  of  effort  on 
the  several  factors  to  the  bidder's  overall  cost  estimate  v/ill 
be  examined.  For  this  purpose,  certain  sum.mary  data  v/ill 
be  required.  These  summary  data  should  be  furnished  as  a 
separate  document  v/hich  should  also  include  a complete  reference 
set  of  the  scope  of  effort  data  sheets. 

E.2  Basic  summary  data  v/ill  include  presentation  of  sub-total 
and  total  cost  estimates  in  standard  format  - direct  costs, 
burden,  materials,  G&A,  fee,  etc. 

E.2.1  These  summ.ary  sheets  should  be  accompanied  v/ith  notes 
reconciling  the  scope  of  effort  data  sheets.  Any  unusual 
data  should  be  explained. 

E.3  If  the  proposal  includes  a substantial  proposed  sub- 
contracting (or  "associate")  cost  figures,  a summary  analysis 
should  be  prepared  and  related  to  the  basic  summary  data. 

E.4  Supporting  schedules  should  be  prepared  detailing  the 
hourly  wage  rate  of  personnel  and  this  related  to  the  basic 
summary  data. 

E.5  Supporting  schedules  should  be  prepared  for  travel  costs 
and  related  to  the  basic  summary  data.  Schedules  should 
include  breakdown  based  on  distance  (or  destination) , duration, 
number  of  travelers,  and  numJDer  of  trips.  Similarly,  supporoing 
schedules  on  telephone  (and  teletype)  by  nummer  of  calls,  and 
average  cost  per  call. 
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Phase  One  Scope  of  Effort 


Note:  The  detailed  organization  parallels  the  organization 

of  the  Statement  of  Work,  Schedule (s),  and  Proposal 
Evaluation  Factors,  reference  to  which  should  be  made 
for  clarification. 

I.  Management 

(Note  that  the  scope  of  work  here  is  limited  to  admini- 
strative or  program  management,  and  should  be  distinguished 

from  effort  properly  reported  under  other  factors.) 

Program  Director  M/M 

Senior  Management  Advisors  M/M 

(Include  general  officers,  or  other  senior 
corporate  personnel,  concerned  v/ith  admini- 
strative management  policy,  if  appropriate) 

Other  (specify)  • M/M 

(Include  comparable  personnel  of  associates, 
subcontractors,  consultants,  if  appropriate; 
include  program  director's  personal  staff, 
if  any) 

Scheduling  M/M 

(Include  costs  of  analysis,  preparation, 
monitoring , etc . ) 

Costing  M/M 

(Include  costs  of  analysis,  preparation, 
monitoring,  etc.) 

Telephone  $ 

(Include  telephone,  teletype  costs,  etc.,  and 
breakdown  by  number  of  calls,  etc.,  if  appropriate) 

Travel  $ 

(Include  breakdown  by  destination,  duration, 
breakdown  by  number  of  trips,  if  appropriate) 

II.  Reports  and  Reviews 

(Note  that  you  should  distinguish  efforts  properly 

attributable  to  other  factors.) 


Liaison  with  ETIP  M/M 

(Include  time  spent  in  conferences  or  visits  both 
at  ETIP  and  contractor's  plant,  or  elsewhere  break- 
down by  number  of  trips,  duration,  number  of 
travelers . ) 


Liaison  with  P7VA 

(Include  on  same  basis.) 

Liaison  with  Others 

(Include  on  same  basis.) 


M/M 

M/M 
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IV. 


V. 


VI. 
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Reports. and  Services 

(Include  time  spent  in  direct  preparation, 
technical  v;riting,  drafting,  etc.,  and 
costs  of  reproductio.n . Ereakdov/n  by  type 
of  report,  etc.,  if  appropriate.) 
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Objectives 

Review  of  objectives  M/M 

(Breakdov;n  between  docume.nts  and  interviews, 
and  among  ETIP,  PAA,  industry,  and  others.) 

t 

Preparation  of  description  and  analysis  M/.M 

(Breakdown  among  ETIP,  PA_A,  i.ndustry,  and 
others . ) 

Preparation  of  preliminary  recomme.ndatio.ns  M./M 

Preliminary  requirements  for  pilot  evaluation  ;*/:* 

system 

Preliminary  requirements  for  prototyce  M/M 

evaluation  system 

Travel  $ 

Background 

Review  of  background  M/M 


(Breakdovm  between  documents 

and  interviews,  and  among  experiments, 

programs  and  organizations.) 


Preparation  of  framework  m.odel  2-1/M 

Travel  $ 

Evaluation  Design 

Preliminary  Design  M/M 

Detailed  Design  M/M 

Preliminary  design  of  pilot  evaluation  system^  M/M 

Preliminary  design  of  prototype  evaluation  system  M/M 

Data  Collection  Process 


(Note  that  you  should  distinguish  data  collections 
which  may  have  been  included  under  prior  factors) 

Design  and  development  of  data  collection  process  M/M 

Data  collection  for  exploratory  evaluations  M/M 
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Design  and  pilot  test  of  instruments  M/M 

Data  collection  for  a priori  proposition  M/M 

testing  evaluations 

Travel  $ 

(Breakdown,  as  appropriate,  between  exploratory 
and  a priori  proposition  testing  evaluations; 
provide,  as  appropriate,  basis  for  estimating 
in  terms  of  trips,  etc.) 

Telephone,  postage,  etc.  $ 

(Breakdown,  as  above.) 

VII.  Data  Analysis  Process 

(Mote  that  you  should  distinguish  from  effort 
included  with  Factor  II.) 

Design  aitd  developm.ent  of  data  analysis  process  M/M 

Exploratory  analysis  and  findings  M/M 

A priori  proposition  testing  analysis  and  findings  M/M 
Analysis  of  evaluation  system,  and  recommendations  M/M 
Analysis  of  Phase  One,  and  recommendations  M/M 
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Phase  Two  and  Three  Scope  of  Effort 

Phase  Two 


It  is  not  expected  that  the  bidder  v/ill  be  able  to  provide  a 
comparable  level  of  detail  for  Phase  T^.-/o . Hcv/ever,  a rough 
estimate  V7ill  be  useful  in  assessing  those  references  to 
Phase  11//0  in  the  proposal,  and  in  planning. 

The  minimum  required  breakdov/n  is  to  distinguish  betv;een  the 
following : 

1.  Those  costs  (labor,  travel,  etc.)  directly  a function  of 
data  collection  a.nd  analysis  for  both  basic  and  special 
data  processes. 

2.  All  of  the  remainder  (management,  design,  etc.) 


Additional  inf orm.ation , consistent  with  these  purposes, 
V70uld  be  appreciated,  but  it  is  not  required. 


Phase  Three 

A similar  distinction  should  be  observed  in  providing  estimates  fo 
Phase  Three.  It  is  expected,  that  these  may  be  m.ore  speculacive , 
but  v/here  there  are  identifiable  assumptions  v.’hich  v;ill  largely 
determine  the  scope  of  effort,  discussion  v;ould  be  helpful. 
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Charles  W.  N.  Thompson,  Associate  Professor 
Technological  Institute,  Northwestern  University 

INTRODUCTION 

The  subject  of  this  paper  is  the  description  of  a procurement  process  developed 
to  meet  a specific  procurement  problem,  and  which  is  currently  in  the  process 
of  being  used  to  procure  two  evaluation  systems  by  the  Experimental  Technology 
Incentives  Program  (ETIP) , National  Bureau  of  Standards,  Department  of  Com- 
merce, The  two  systems  (see  Figure  I which  outlines  the  design  process  each 
will  go  through  in  parallel)  are  to  provide  an  on-going  capability  to  evaluate 
the  agency  impact  and  commercial  impact,  respectively,  of  a series  of  procure- 
ment experiments  initiated  in  the  Procurement  Policy  Area  of  ETIP  and  being 
carried  out  in  cooperation  with,  and  under  the  direction  of,  several  govern- 
mental procuring  agencies.  These  experiments,  which  are  not  the  direct  subject 
of  this  paper,  are  designed  to  test  specific  improvements  in  the  methods  used 
by  governmental  agencies  to  procure  products,  with  specific  emphasis  on  methods 
which  will  increase  the  utilization  of  new  and  improved  technology  and  achieve 
economic  savings  and  better  performance.  The  procurement  process  to  be  des- 
cribed here  was  not  part  of  the  set  of  procurement  experiments  but  has  become 
an  "unintended  experiment"  itself. 

This  paper  is  the  first  published  report  (known  to  this  writer)  which  describes 
the  process,  although  it  was  first  developed  and  used  nearly  twenty  years  ago. 
In  the  sections  which  follow,  a brief  historical  background  is  furnished,  fol- 
lowed by  two  sections  describing  the  process,  a discussion  of  the  salient 
characteristics  of  the  process,  applications  of  the  process,  and  related 
theory. 

The  generic  reference  to  the  "procurement  process"  reflects  one  of  the  unsolved 
problems,  that  of  giving  it  a descriptive  name.  The  earliest  references,  when 
a name  was  required,  were  a choice  among  "Project  Level  Systems  Management," 
"Proposal  Preparation  Procedure,"  or  the  "Golden  Fleece,"  a reference  to  the 
system  to  be  procured.  More  recently  the  name  "Parametric  Factor  Analysis" 
was  used,  but,  while  technically  accurate,  it  uses  terms  which  have  a differ- 
ent, and  specialized,  technical  meaning  in  a major  discipline.  Similarly,  a 
number  of  other  potentially  descriptive  terms  (multi-attribute,  discrete,  in- 
dependent, ranking,  etc.)  present  problems.  Recently  a colleague  of  the  writer 
referred  to  the  process  as  "Thompson's  Method"  for  want  of  a more  convenient 
handle,  but  the  problem  remains. 


BACKGROUND 

One  of  the  major  periods  in  the  development  and  application  of  methods  for  the 
procurement  of  large-scale  weapons  systems  (and,  later,  space  systems)  began  at 
Wright  Field  during  the  early  1950's  with  the  introduction  of  the  "weapons  sys- 
tems management  concept"  as  an  alternative  to  the  parallel  development  under 
the  direction  of  functional  laboratories  of  the  basic  aircraft  and  its  compo- 
nent "black  boxes."  The  preference  for  "project  management"  over  "functional 
management"  was  centered  on  the  need  to  solve  the  problems  of  coordinating  the 
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scheduling  of  development  and  achieving  technical  interface.  During  this  same 
time  there  was  a growing  recognition  that  the  aircraft  was,  itself,  a "black 
box"  part  of  a "system"  designed  to  carry  out  a specific  function  or  mission. 
From  this  point  of  view,  any  specific  part  ("black  box")  could  be  considered 
part  of  one  or  more  systems,  often  overlapping  (e.g.,  weapon  system,  logistics 
system,  reconnaissance  system,  tactical  or  strategic  warfare  system,  etc.;,  and 
should  be  designed  (and  procured)  with  consideration  of  its  relation  to  the 
several  systems  of  which  it  would  be  part.  In  the  Electronic  Reconnaissance 
Section,  Aerial  Reconnaissance  Laboratory,  the  project  engineers  responsible 
for  development  of  airborne  equipment  to  intercept  and  analyse  electronic  sig- 
nals from  potentifl  enemy  radars  and  other  emitters  proposed  to  design  and  pro- 
cure systems  (called  "subsystems")  Instead  of  "black  boxes"  but  were  faced  with 
several  problems.  First,  Che  requirements  (or  specifications)  were  constantly 
changing  because  of  the  need  to  respond  Co  new  developments  in  radar  (and  other 
emitters).  Second,  few  laboratory  personnel  had  any  significant  prior  ex- 
perience in  designing  or  procuring  systems.  And,  finally,  equipment  suppliers 
had  little  relevant  experience  (except,  in  some  cases,  as  subcontractors  to 
systems  contractors).  In  this  context,  the  procurement  process  to  be  described 
was  developed  and  applied,  initially,  to  two  electronic  recorj.aissance  and  one 
weather  reconnaissance  subsystems.  The  original  procurement  exhibits  were  sub- 
sequently extended  and  incorporated  into  a manual  for  procurement.  [Iw] 

The  application  of  this  procurement  process,  developed  for  military  hardware 
subsystems,  to  the  procurement  of  an  evaluation  (software)  capability  was 
initiated  about  six  months  ago  when  the  apparent  similarity  of  the  situations 
(and  the  problems)  was  first  recognized.  First,  the  evaluation  requirements 
t?ere  not  only  constantly  changing  because  of  the  on-going  progress  in  the 
initiation  and  carrying  out  of  procuretcent  experiments  but  also  included  a vide 
variety  of  objectives  ranging  from  very  specific  to  very  broad,  and  from  im- 
mediate to  long-range.  Second,  ETI?  had,  with  the  exception  of  the  writer, 
little  or  no  experience  in  systems  procurement,  and  neither  the  time  nor  per- 
sonnel to  accomplish  a problem  or  concept  definition  design.  Finally,  it  did 
not  appear  that  the  "evaluation  industry"  had  any  significant  present  capabil- 
ity' to  design  or  manage  an  "evaluation  system"  to  meet  ETIP  requirements. 

Based  upon  this  opportune  coincidence,  the  development  of  the  procurement  pack- 
age and  the  organization  of  the  procurement  was  accotcpl ished  with  the  close, 
and  progressive  cooperation  of  personnel  of  ETIP,  the  contracting  office,  the 
program  administrative  agencies,  and  others.  In  the  next  two  sections,  the 
procurement  process  is  described  in  outline  form  in  two  parts.  First,  the  de- 
tail development  of  the  entire  procurement  process,  which  is  incorporated  into 
the  "Proposal  Preparation  Procedure"  [15]  and  associated  documents.  Second, 
a description  of  the  proposal  evaluation  process  itself,  which  is  incorporated 
into  the  "Proposal  Evaluation  Procedure."  [15] 

PROPOSAL  PREPARATION  PROCEDURE 

The  initial  step,  upon  recognition  that  this  procurement  process  is  appropriate, 
is  to  review  what  you  "know*'  about  the  procurement.  This  is  a kind  of  prelim- 
inary design  or  concept  or  problem  analysis,  and  is  in  the  form  of  the  identi- 
fication or  listing  of  parametric  factors  (those  conditions  or  limits  which 
help  to  define  wliat  it  is  you  are  going  to  procure).  These  should  include,  for 
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example : 

a)  an  overall,  if  rough,  schedule  of  the  time  available  to  let  the  con- 
tract, any  early,  specific  outputs  desired,  beginning  and  ending  of 
phases  (if,  for  example,  the  program  can  be  broken  down  into  phases), 
key  milestones,  and  the  expected  completion  date; 

b)  a comparable  cost  estimate,  broken  down,  if  possible,  into  alloca- 
tions by  phase,  by  major  types  of  activity  (e.g.,  study  and  design 
versus  field  data  collection),  or  other  rough  divisions  of  effort; 

c)  a preliminary  outline  description  of  what  you  want,  preferably  in 
terms  of  the  kinds  of  things  you  believe  need  to  be  ‘accomplished 
throughout  the  life  of  the  program  (process  specification)  rather 
than  specific  end  products  (or  services) ; 

d)  a preliminary  identification  of  reference  documents  which  may  be 
useful  to  you  (and,  eventually,  to  the  bidders)  in  further  de- 
fining the  program;  and,  finally,  and  perhaps  one  of  the  most 
important  areas, 

e)  Identification  of  the  key  personnel  who  are  presently  concerned 

with  the  success  of  the  program  and  who  will  be  actively  concerned 
in  the  future  (and  who  will  provide  the  basis  for  the  evaluation 
team) , , 

The  next  step  is  to  refine  the  above  set  of  parametric  factors  in  the  form  of 
a series  of  statements  describing  the  characteristics  of  a prospective  bidder/ 
contractor  (in  terms  of  understanding  of  the  problem  and/or  capability  to  per- 
form) which  you  believe  critical  to  the  successful  carrying  out  of  the  program. 
This  can  begin  by  identifying  major  areas  of  concern  or  problem  areas  (e.g., 
management,  program  objectives,  major  technical  problems  or  skill  areas)  or  by 
listing  specific  critical  problem  areas  (e.g.,  schedule  and  cost  control,  ETIP 
objectives,  data  collection  design).  In  this  process  reference  can  be  made  to 
the  results  of  the  previous  stage,  or  the  progressive  drawing  upon  contri- 
butions from  individuals  who  will  be  members  of  the  evaluation  team,  from  pro- 
gram documents  (Including  organizational  descriptions  of  the  objectives  of  the 
agency  or  of  the  specific  program),  and  from  other  sources,  such  as  prior  pro- 
posal evaluation  procedures.  The  specific  concern  is  to  obtain  statements  in 
the  form  of  critical  specific  characteristics.  It  is  desirable  that  these 
statements  minimize  overlap  and  gap,  but  this  is  not  a critical  requirement. 
For  convenience  of  both  the  bidders  and  the  evaluation  team,  the  number  of  in- 
dividual statements  (of  specific  critical  problem  areas)  should  range  from  no 
less  than  twenty  to  no  more  than  seventy.  These  individual  statements  (re- 
ferred to  hereafter  as  sub-factors)  should  then  be  grouped  into  (approximately 
five  to  fifteen)  major  areas  of  concern  (hereafter  referred  to  as  factors) 
which  reflect  related  problems  (and  related  interests  and  skills  on  the  part 
of  evaluation  team  members).  Because  these  factors  (and  sub-factors)  will 
provide  the  central  framework  for  the  preparation  of  proposals  and  for  the 
evaluation  team,  it  is  important  that  the  focus  be  clear;  for  this  purpose, 
additional,  specific,  Illustrative  questions  can  be  added  to  each  sub-factor. 
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An  illustration  of  the  results  of  this  process  is  provided  in  the  three  tables 
at  the  end  of  this  paper. 

Using  this  framework  of  factors  (and  sub- factors) , the  "Statement  of  Work"  [15] 
and  "Schedule"  [15]  are  prepared;  this  serves  as  a double-check  on  wiiether  all 
the  significant  parts  of  the  program  are  covered.  The  final  key  document  is 
called  "Scope  of  Effort"  [15],  and,  in  parallel  with  the  above,  it  provides  a 
framework  for  the  bidder  to  present  his  estimate  of  how  he  will  allocate  man- 
power and  dollars  to  the  various  parts  of  the  program.  If  the  program  is  to 
be  divided  into  time  phases,  this  may  be  reflected,  as  required,  in  the  organ- 
ization of  the  above  documents.  To  these  may  be  added  copies  of  program  les- 
criptions,  descriptive  or  amplifying  notes,  and  any  other  documents  which 
might  be  helpful  to  the  bidders. 

A summary  description  of  the  above,  and  detailed  guidelines  for  the  preparation 
of  proposals,  is  incorporated  in  the  "Proposal  Preparation  Procedure"  [15], 
which,  together  with  a description  of  the  sub-factors  and  factors  incorporated 
in  a separate  document  called  the  "Proposal  Evaluation  Factors"  [15],  and  the 
other  documents  noted  above,  makes  up  the  procurement  package  which  is  append- 
ed to  the  formal  part  of  the  Request  for  Proposal.  Where,  as  in  the  present 
case,  the  procurement  process  represents  a significant  departure  from  prior 
practice,  it  is  considered  desirable  to  hold  two  bidders'  briefings,  the  first 
to  allow  prospective  bidders  to  decide  whether  or  not  to  bid  and  to  prepare 
questions  which  can  be  answered  at  the  second.  For  the  same  reason,  and  for 
the  additional  purpose  of  increasing  openness,  communication,  and  confidence, 
the  "Proposal  Preparation  Procedure"  [15]  also  describes  briefly  how  and  why 
the  process  was  developed  (where  we  are  coming  from)  and  how  the  proposal 
evaluation  will  be  carried  out  (where  we  are  going). 

PROPOSAL  EVALUATION  PROCEDURE 

The  broad  outlines  of  the  procedure  to  be  used  for  proposal  evaluation  are  in- 
cluded in  the  "Proposal  Preparation  Procedure"  [15]  described  above  and  which 
is  provided  to  the  bidders.  In  addition,  members  of  the  evaluation  team  re- 
ceive a more  detailed  set  of  guidelines  incorporated  in  the  "Proposal  Evalua- 
tion Procedure"  [15],  (Note:  At  the  time  of  this  writing,  the  second  bidders' 

briefing  ha?  been  completed  but  proposals  have  not  been  received;  the  descrip- 
tion which  follows  is,  therefore,  based  upon  the  preparation  for  evaluation  on 
this  procurement,  and  in  the  perspective  of  prior  experience.)  The  members  of 
the  evaluation  team  are  drawn,  as  much  as  possible,  from  the  set  of  individuals 
who  will  be  concerned  directly  with^  the  successful  carrying  out  of  the  program 
which  results  from  the  award;  where  particular  background  or  skills  are  neces- 
sary to  the  evaluation  of  a particular  factor,  others  may  be  included.  In  all 
cases,  membership  is  limited  to  those  who  accept  responsibility  for  evaluating 
at  least  one  sub-factor.  The  steps  of  the  process  include  the  following: 

a)  individual  preparation,  in  which  members  of  the  team  become  familiar 
with  the  entire  RFP , receive  copies  of  all  of  the  key  documents  (and 
have  access  to  all  of  the  reference  documents) , and  are  invited  to 
participate  in  the  briefings  of  bidders; 
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b)  prelirainary  review,  by  individual  members,  of  the  proposals  after 
receipt,  and  preparation  of  preliminary  ratings  (on  at  least  spe- 
cific, assigned  sub-factors)  from  "Nothing  can  stop  him..."  to 
"No  hope  (totally  unacceptable)"  which  will  become  part  of  the 
record ; 

c)  a aeries  of  sequential  (factor  team)  meetings  at  wiiich  individuals 
responsible  for  the  set  of  sub-factors  within  each  factor  agree  on 
a single,  aggregate  rating  of  each  bidder  for  each  factor; 

d)  the  preparation  of  a sunnnary  of  all  of  the  factor  ratings,  to- 
gether with  a summary  description,  for  final  review,  negotiation 
with  the  successful  bidder,  and  award. 

There  are  few  restrictions  on  the  judgment  process  of  the  individual  team  mem- 
ber; he  may  use  prior  information  (but  must  disclose  it  to  other  team  members) ; 
he  must  read  the  parts  of  the  proposals  directed  to  the  sub-factors  to  which 
he  is  assigned,  but  he  may  read  (and  use)  any  or  all  other  parts;  he  will  know 
the  identity  of  the  bidders  and,  for  that  matter,  will  have  access  to  any  in- 
formation any  other  member  of  the  team  has;  he  is  required  to  put  the  results 
of  his  individual  review  in  the  record,  but  he  may  change  his  mind  during  the 
review  by  the  factor  team;  and  he  may  attend  meetings  of  factor  teams  of  which 
he  is  not  a part. 

While  individuals  are  encouraged  to  do  so,  the  factor  team  must  rate  each  sub- 
factor, and  the  overall  factor,  by  considering  the  bidder's  comprehension  or 
understanding  of  the  problem  presented,  approach  described  or  outlined,  and 
present  or  potential  capability  to  perform.  They  may  cross-reference  other 
parts  of  the  proposal,  and  may  draw  upon  the  "Scope  of  Effort"  [15]  informa- 
tion as  a check  on  what  the  bidder  expects  to  do.  There  is  no  formal  voting 
procedure,  and  no  prescribed  comparative  weighting  of  the  sub-factors;  they 
may  inquire  into  the  basis  for  the  position  of  a specialist  in  his  area,  they 
may  record  strong  individual  disagreements,  and  they  are  encouraged  to  add 
specific  comments  which  would  be  useful  in  negotiation.  Each  factor  team  must 
continue  until  it  has  reached  an  agreement,  which  is  then  recorded  on  the  same 
form,  and  with  the  same  scale,  that  the  members  used  in  their  individual  re- 
view, Because  these  guidelines  are  critical  to  the  factor  team  process,  they 
appear  not  only  in  written  form  but  are  also  reviewed  (or  even  read  aloud)  at 
the  beginning  of  each  factor  team  meeting. 

Ope  of  the  few  significant  changes  from  the  procedure  as  used  previously  is 
the  substitution  of  written  questions  for  an  oral  presentation.  In  the  origi- 
nal procedure,  each  bidder  was  invited  to  make  a brief  oral  presentation  and 
to  answer  specific  questions  (both  of  which  were  recorded)  from  the  evaluation 
team  in  order  to  assure  that  the  team  correctly  understood  what  the  bidder  pro- 
posed and  had  sufficient  information  upon  which  to  base  a confident  rating 
(whether  in  favor  of  or  against  the  bidder).  Because  of  the  uncertainty  of  the 
number  of  bidders  in  the  present  procurement  and  insufficient  time  (if  there 
were  a large  number) , the  procedure  was  changed  to  provide  that  members  of  the 
team  could  direct  specific,  written  questions  (through  the  Contracting  Officer) 
to  an  individual  bidder  if  necessary  to  clarify  an  ambiguity  or  to  otherwise 
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make  a confident  rating  possible. 

COMMENTS  ON  TliE  PROCESS 

The  process  described  in  outline  form  above  has  a number  of  features  wfiich, 
taken  separately,  probably  are  not  novel  or  unique.  The  process  as  a whole, 
however,  is  considered  a significant  departure  from  present  methods  of  procur- 
ing program  evaluations  (and  evaluation  systems  or  capabilities),  and,  at  the 
time  of  its  initial  use  for  military  hardware  subsystems  procurement,  was  a 
significant  departure  from  even  the  then  developing  weapons  systems  procure- 
ment methods.  In  the  paragraphs  which  follow  some  of  the  salient  character- 
istics of  the  process  will  be  identified  or  described,  noting  whether  they  are 
required  or  desired  characteristics,  w’hat  the  alternatives  are,  and  wh.y  this 
approach  was  used,  as  appropriate.  Overall  characteristics  will  be  discussed 
first,  followed  by  characteristics  of  specific  parts  of  the  process. 

The  first  overall  characteristic  is  that  the  process  is  designed  to  meet  a par- 
ticular kind  (or  class)  of  procurement  problem(s) , in  terms  of  the  procuring 
agency,  the  set  of  prospective  suppliers  (bidders/contractors),  and/cr  the  kind 
of  thing  to  be  procured.  No  claim  is  made  that  the  process  is  applicable 
everywhere  (e.g.,  advertised  procurements),  and  this  characteristic  will  be 
discussed  further  in  the  next  section  of  this  paper  on  "Applications." 

The  second  overall  characteristic  is  that  the  process  is  essentially  pragmatic. 
It  proposes  that  uncertainty  be  faced  and  not  avoided,  that  where  we  know  a lot 
about  V7hat  we  want  that  we  use  that  knowledge,  that  where  ve  are  dot  sure  or  do 
not  know  that  we  disclose  the  state  of  our  knowledge  and  try  to  develop  the 
"best"  description  (which  is  likely  to  be  a process  rather  than  end  product 
description)  we  can.  And  it  proposes  that  we  disclose  this  to  prospective  bid- 
ders, so  that  they  can  concentrate  on  providing  us  the  "best"  information 
available  for  the  next  step  in  the  process--the  selection  of  the  contractor  who 
will  work  with  us. 

The  third  general  characteristic  is  that  it  is  an  open,  explicit,  cooperative 
process,  recognizing  the  crucial  importance  of  improving  communication  (both 
the  content  of  the  information  shared  and  confidence  in  that  information)  among 
the  several  parties.  Prospective  bidders  are  provided,  as  much  as  possible, 
with  all  of  the  information  available  (with  the  narrow  exception  of  that  barred 
by  law  or  regulation,  and,  of  course,  the  evaluation  team  deliberations  while 
they  are  going  on) , and  both  the  bid  set  and  the  selection  of  reference  docu- 
ments are  designed  to  facilitate  their  ability  to  prepare  proposals  (and  to 
have  confidence  in  the  process).  Similarly,  the  evaluation  team  (which,  where 
possible,  includes  representatives  of  other  agencies,  such  as  "using  agencies," 
with  a significant  interest  in  the  success  of  the  procurement,  and  others  who 
may  be  interested  in  the  process)  has  access  to  all  of  the  information  avail- 
able, and  each  member  is  assured  that  his  part  in  the  process  is  preserved  in 
the  record  and  in  the  selection  of  the  winning  bidder. 

The  most  salient  specific  characteristic  of  the  process  is  the  method  of  iden- 
tifying and  describing  the  factors  and  sub-factors  wiiich  provide  the  basis  for 
preparing  proposals  and  for  evaluating  proposals.  The  process  neither  assumes 


174 


« 


451 


nor  requires  that  any  formal  structure  or  framework  exists  for  the  object  of 
the  procurement;  to  the  extent  there  is  structure,  it  can  be  used,  but  the 
process  is  not  dependent  upon  a complete  and  balanced  description,  or  a set  of 
Inclusive  and  exclusive  parts.  Instead,  the  process  treats  each  sub-factor  as 
a separate  "probe,”  obtaining  a measure  along  a dimension  (or  viewpoint)  of  the 
whole  program  (or  whole  capability  of  the  bidder),  a measure  which  may,  to  some 
degree,  overlap  or  intersect  some  other  measure  (sub-factor) , and,  when  com- 
bined with  all  of  the  other  measures,  is  intended  to  provide  a set  of  measures 
which  describe  the  program  (or  the  capability  of  the  bidder).  The  several  ad- 
vantages of  this  include  the  following: 

a)  it  is  not  required  that  all  parts  of  the  system  be  described  or 
specified  to  the  same  degree  of  detail,  each  part  can  be  described 
to  the  extent  its  description  is  known,  parts  which  depend  upon  the 
prior  definition  of  some  other  part  can  be  defined  as  such,  and 
even  inconsistent  parts  (e.g.,  statements  of  objectives  prepared  at 
different  times  for  different  purposes  by  different  people)  can  be 
included ; 

b)  it  is  not  required  that  there  be  a major  preoccupation  with  assuring 
that  each  factor  (or  sub-factor)  be  completely  independent  (and  non- 
overlapping) with  respect  to  others,  allowing  a concentration  on 
••what  it  measures”  rather  than  trying  to  avoid  duplication; 

c)  it  is  possible  to  start  any%^ere,  to  jump  around,  to  repeat  and  re- 
work individual  sub-factors  (and  factors),  a process  which  "builds 
up"  to  the  whole,  and  recognizes  that  the  set  may  not  fit  neatly 
into  any  single,  explicit,  formal,  external  framework. 

With  considerable  oversimplification,  the  above  approach  taay  be  contrasted  with 
several  other  approaches.  In  the  procurement  of  research  (and,  in  some  cases, 
of  professional  services) , there  may  be  a comparably  high  degree  of  uncertainty 
on  the  part  of  the  procuring  agency  (and  even  the  prospective  supplier),  but  it 
may  be  reasonable  to  limit  the  evaluation  to  a "single"  or  small  set  of  tech- 
nical problems,  together  with  an  evaluation  of  the  capability  of  the  individ- 
ual (a)  who  will  work  on  the  project;  similarly,  but  at  the  other  end  of  the 
uncertainty  spectrum,  a production  procurement  to  detailed  specifications  may 
reasonably  be  evaluated  on  the  "single"  question  of  price,  together  with  an 
evaluation  of  the  facilities  needed  to  perform.  Some,  if  not  many,  of  the 
proposal  evaluation  boilerplates  appear  to  follow  this  binary  breakdown  between 
a single  (or  small  set  of)  selected  major  characteristic (s)  and  a "lumped" 
assessment  of  capability  (of  personnel  and/or  the  organization)..  The  forced 
selection  of  a (usually)  small  number  of  technical  problems  may  result  in  an 
overemphasis  on  some  problems  (and  ^^ich  may  Include  problems  which  cannot  be 
adequately  described  or  "solved"  within  the  bidding  period)  and  the  exclusion 
of  others,  perhaps  equally  critical,  which  the  bidder  may  feel  compelled  to 
respond  to  because  of  prior  knowledge  or  from  "hints"  he  receives  from  other 
descriptive  materials,  e.g.,  program  plans  provided  as  part  of  the  bid  set. 
Coupled  with  this  is  the  difficulty  of  assigning  a single,  overall  measure  to 
the  "qualifications"  of  the  bidder,  either  as  an  Independent  assessment  or  one 
which  parallels  the  technical  part.  For  at  least  some  of  these  "middle-ground" 
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procurements,  the  additional  flexibility  and  freedom  to  define  the  evaluation 
in  terms  of  whatever  factors  can  be  identified  would  appear  to  be  desirable. 

The  next  specific  characteristic,  and  related  to  the  above,  is  the  use  of  a 
single  framework,  the  set  of  "Proposal  Evaluation  Factors"  [15],  for  the  prepa- 
ration of  proposals  and  for  their  evaluation.  3y  organizing  the  "Statement  of 
Work,"  the  "Schedule,"  and  the  "Scope  of  Effort"  [15]  on  the  same  basis,  and 
making  clear  that  all  other  descriptive  materials  are  relevant  only  to  the  ex- 
tent they  contribute  to  the  central  framework,  the  bidders  and  the  evaluation 
team  have  a common  frame  of  reference.  The  bidder  can  concentrate  his  atten- 
tion on  providing  information  which  will  describe  his  capability  (understand- 
ing of  the  problem,  present  or  proposed  approach,  and  present  or  potential 
capability  to  perform)  with  respect  to  a specific  set  of  sub-factors;  the 
evaluation  team  can,  likewise,  concentrate  its  attention  on  the  same  set,  and 
with  a minimum  risk  of  overlooking  the  relevant  part  of  the  proposal.  They 
share,  to  a large  degree,  a conmon  set  of  information  and  know  the  rules. 

A related  specific  characteristic  is  the  elimination  of  the  necessity  for  a 
limitation  on  length.  The  evaluation  team  member  can  quickly  find  t.ne  infor- 
mation relevant  to  a specific  sub-factor,  and  the  bidder  can  be  sure  that  what 
he  is  presenting  will  be  found;  each  can,  if  necessary,  develop  or  quickly 
identify  necessary  cross-references  to  other  parts  of  the  proposal.  The  ne- 
cessity (or  desirability)  for  over-long,  "shotgun"  approaches  to  cover  any 
conceivable  area  which  may  be  of  interest  to  some  individual  member  of  the 
evaluation  team,  or  to  cover  ambiguities  in  the  amount -of  detail  required,  is 
diminished;  in  contrast,  there  is  an  obvious  premium  on  clarity  and  conciseness 
in  zeroing  in  on  the  specific  sub-factor. 


The  next  specific  characteristic  is  the  makeup  of  the  evaluation  team.  Instead 
of  centering  the  responsibility  (and  burden)  for  the  entire  evaluation  on  the 
project  officer  (together  with  anyone  he  can  trap  into  helping  him),  the  team 
is  formed  by  calling  upon  a wider  set  from  among  those  who  have  a prior  (and 
continuing)  interest  in  the  subject  of  the  procurement  or  who  have  special 
competence  to  evaluate  particular  sub-factors  (and  factors).  Because  of  the 
framework  used,  participation  in  the  evaluation  process  can  be  limited,  by  the 
individual,  to  reading  parts  of  the  proposal,  evaluating  those  parts,  and  par- 
ticipating in  only  those  factor  team  meetings  concerned  with  those  parts.  This 
functional  breakdown  substantially  limits  the  time  commitment,  and  it  allows 
concentration  on  parts  which  are  of  specific  interest  (and/or  within  the  spe- 
cific competence)  of  the  individual,  A larger  evaluation  team  provides  an  op- 
portunity for  those  interested  in  parts  of  the  procurement  (and  who  may  be 
called  upon  to  work  with  the  successful  contractor  in  the  future)  to  provide 
inputs  and  to  contribute  their  expertise  and  perspective;  and,  similarly,  spe- 
cialists in  particular  areas  can  increase  the  likelihood  of  an  effective  eval- 
uation. In  a systems  sense,  these  team  members  provide  an  interface  with  the 
several  systems  they  represent. 
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may  be  useful  In  a research  procurement,  the  object  here  is  to  rate  the  spe- 
cific sub-factor  in  the  light  of  all  the  relevant  evidence,  and  this  requires 
access  to  the  whole  proposal,  if  desired.  Similarly,  it  is  considered  only 
realistic  to  recognize  that  individual  members  of  the  evaluation  team  bring  a 
wide  variety  of  prior  experience  (and  information)  not  only  on  areas  relevant 
to  the  several  sub-factors  but  also,  in  some  cases,  with  respect  to  the  capa- 
bilities of  specific  bidders.  While  this  information  may  either  add  to  or 
aubtract  from  the  rating  of  the  specific  bidder,  there  is  no  advantage  to  pre- 
tending it  will  be  not  considered,  and  its  weight  is  more  properly  assessed, 
and  accounted  for,  by  the  evaluation  team  through  a requirement  that  it  be 
openly  disclosed. 

The  next  characteristic  is  a corollary  of  the  above.  The  members  of  the  eval- 
uation team  are  instructed  that  they  must  rate  each  bidder  affirmatively, 
t.e, , they  must  have  a basis  for  low  ratings  as  well  as  high  ratings.  Where 
there  is  ambiguity  or  an  unclear  or  missing  response,  the  evaluation  team  can- 
not assign  a "zero"  but  must  obtain  the  required  information  by  asking  a spe- 
cific question.  It  is  recognized  that  in  some  cases  of  an  extremely  (and  uni- 
formly) weak  bidder,  the  team  may  properly  decide  that  the  proposal  is  a clear 
and  unambiguous  representation  of  the  capability  of  the  bidder. 

The  final  specific  characteristic  to  be  noted  is  the  process  for  "aggregating" 
the  individual  ratings  of  each  team  member  with  the  ratings  of  the  other  team 
members,  and.  In  turn,  "aggregating"  sub-factor  ratings  into  factor  ratings, 
and,  in  turn,  into  an -overall  rating.  The  process  is  sequential,  with  a sum- 
mary hard-copy  record  retained  of  each  stage  to  provide  a traceable  record  of 
the  process  and  as  additional  assurance  that  the  contributions  of  the  individ- 
ual team  members  are  reflected  in  the  end  result.  The  basis  for  aggregation  of 
Individual  team  member  ratings  and  of  the  sub-factor  ratings  is  the  factor 
team;  using  any  process  acceptable  to  them,  the  members  of  the  factor  team  de- 
termine the  relative  rating  of  all  of  the  bidders  against  a fixed  scale  (the 
same  one  each  used  for  individual  ratings  on  individual  oub-factors) . Formal 
voting  is  precluded;  somewhat  analogous  to  a petit  Jury  or  an  arbitration 
team,  the  factor  team  is  required  to  interchange  information  and  explore  their 
several  positions  until  they  arrive  at  a mutually  acceptable  result.  Provi- 
sions are  made  to  append  clarifying  or  qualifying  notes,  and,  particularly,  to 
identify  areas  which  will  require  negotiation  (or  further  clarification)  in  the 
event  a particular  bidder  is  chosen.  In  prior  experience  with  nearly  forty 
factor  teams  there  were  no  instances  in  which  this  was  not  accomplished.  The 
final  step  of  aggregation,  the  combining  of  the  ratings  across  all  of  the  sepa- 
rate factors,  is  the  first  one  in  which  formal  weights  are  introduced,  and 
these,  characteristically,  are  limited  to  a range  of  no  more  than  two  to  one 
(and  this  is  disclosed  to  the  bidders).  This  step  is  largely  mechanical,  and 
the  summary  report  will  not  only  Include  a description  of  the  strong  and  weak 
points  of  the  several  bidders  but  also  be  accompanied  with  the  complete  set  of 
factor  ratings.  While  the  summary  weighted  score  nay  serve  to  identify  the 
"winner"  and  other  acceptable  bidders,  the  individual  factor  ratings  direct  at- 
tention to  any  relative  weak  points.  Similarly,  the  summary  score  and  factor 
ratings  will  identify  "in-between"  and  weak  bidders,  and  specific  areas  of 
unacceptabtllty . 
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If  the  process  can  be  captured  In  terms  of  simple  underlying  concepts,  there 
are  at  least  three  which  would  appear  to  be  central.  First,  where  uncertainty 
(of  various  kinds)  makes  unlikely  (or  difficult)  the  identification  of  wriat, 
in  research  terras,  might  be  referred  to  as  the  dominant  independent  variable 
(or  ’’summing  independent  variable"),  the  set  of  parameters  (plausible  rival 
hypotheses)  are  used  as  a surrogate  independent  variable.  In  contrast  to  the 
more  familiar  (research)  approach  in  a priori  proposition  testing,  wnere  pa- 
rameters are,  at  best,  "unwanted"  variables,  to  be  controlled  or  otherwise 
isolated  from  the  central  problem  of  interest,  in  this  process  we  "use"  the 
parameters  tc  increase  the  likelihood  that  we  shall  achieve  our  (administra- 
tive) purpose  of  obtaining  some  desired  value  of  the  dependent  variable. 
Second,  the  process  is  structured  sequentially  to  provide  for  a series  of 
decompositions  and  compositions  in  several  dimensions,  including  a degree  of 
"value  free"  partitioning.  Third,  the  process  is  explicit,  open,  relatively 
fully  defined  and  self-checking,  including  the  sense  of  "rule  of  law." 

APPLICATION’S  OF  THE  PROCESS 

As  has  been  noted,  this  procurement  process  v;as  developed  to  meet  a specific 
procurement  problem  and  may  be  neither  needed  for  nor  applicable  to  otner 
specific  procurement  problems.  It  would  appear  to  be  inapplicable  to  procure- 
ments in  which  there  is  a single  (or  small  number  of)  critical  basis  (es)  for 
evaluation,  e.g.,  procurements  for  vrell-def ined  products  or  seir.’ices  where  the 
basis  is  price,  and  research  procurements  where  the  basis  is  a particular 
technical  approach  or  capability.  More  generally,  there  would  appear  to  be  no 
need  for  this  procurement  process  where  the  prior  experience  of  the  procuring 
agency  (and  the  set  of  prospective  suppliers)  has  provided  a basis  for  suc- 
cessful procurement and  especially  where  there  are  comparable  previous  pro- 
curements. There  are  probably  a number  of  other  procurement  situations  in 
which  alternative  methods  are  being  used,  including  the  following; 

a)  where  the  procuring  agency  has  the  time  and  resources  to  carry  out 
the  preliminary’  design  phase  (or  even  further  stages)  and  prepare 
detailed  specifications; 

b)  where  the  procuring  agency  has  the  money  (and  time)  to  let  parallel 
(competitive)  contracts  for  the  preliminary  design  (or  concept) 
phase ; 

c)  where  the  procuring  agency  has  the  time  (and  money)  to  let  a "sole 
source"  contract  for  the  preliminary  design  phase,  with  the  flex- 
ibility to  drop  the  program  or  to  start  over  if  the  first  effort 
was  unsuccessful; 

d)  where  one  or  more  prospective  suppliers  are  willing  and  able  to 
carry  out  the  preliminary  design  phase  (or  even  further  phases) 
prior  to  action  by  the  procuring  agency; 

e)  where  the  item  to  be  procured  can  be  "decomposed"  from  technical 
and  scheduling  interfaces  through  the  use  of  standards,  and  the 
like,  or  where  there  are  acceptable  alternates  available. 
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With  these  exceptions  noted,  it  should  not  be  assumed  that  this  procurement 
process  is  limited  to  large  and  complex  systems,  whether  hardware  or  software. 
The  basic  criteria  for  use  of  this  process  include  the  following: 

a)  that  there  is  a degree  of  uncertainty  or  complexity  in  the  item  to 
be  procured  which  makes  it  difficult  to  describe  the  item  to  bidders 
and  to  identify  a suitable  basis  for  evaluating  bidders*  proposals, 
using  available  procurement  methods; 

b)  that  there  are  limits  on  tine,  money,  or  personnel  which  preclude 
considering  other  alternatives;  and 

c)  that  there  is  some  basis  in  prior  experience  for  believing  that 
it  is  critical  to  both  the  procuring  agency  and  the  (successful) 
bidder  to  develop  a reasonably  sound  initial  mutual  understanding 
in  order  to  ensure  a progressive  and  successful  mutual  accorrplish- 
ment  of  the  object  of  the  procurement. 

A maaber  of  possible  applications  might  be  suggested,  and  the  categories  which 
follow,  and  the  illustrations,  represent  only  one  way  in  which  they  might  be 
organized.  First,  the  case  of  multiple  candidates  (bidders  or  projects)  which 
must  be  compared  against  an  uncertain  or  ccxnplex  set  of  requirements,  includ- 
ing: 

a)  con?»etitlve  procurement  based  on  a complex  set  of  design  requirements 
or  of  a system,  e.g.,  the  present  case; 

b)  rank  ordering  of  a series  of  competing  projects  (which  may  not  be 
directly  comparable)  against  the  twin  standards  of  likelihood  of 
success  and  probable  payoff  (future  value)',  i.e.,  the  ’’project  se- 
lection" process; 

c)  comparison  of  alternative  techniques,  solutions,  or  (assumed)  resources 
end  constraints  against  a set  of  "independent"  and/or  differentially 
affected  requirements,  e.g.,  cost-benefit  analysis. 

Second,  the  case  of  a single  candidate  (bidder  or  project)  which  must  be  com- 
pared against  a complex  or  uncertain  set  of  requirements,  including: 

a)  a project  or  proposal  submitted  for  "acceptance"  under  a complex 
statutory  requirement,  e.g.,  application  for  qualification  as  a 
health  maintenance  organization  (HMO) , submission  of  an  "impact 
statement"  for  a power  plant  or  other  facility; 

b)  comparison  of  a proposed  project  against  a set  of  "independent"  re- 
quirements, e.g.,  cost-benefit  analysis. 

Third,  Che  case  where  one  (or  more)  candidates  are  to  be  compared  against  one 
(or  more)  uncertain  or  complex  requirements  by  a set  of  raters  v»ho  may  differ 
not  only  in  the  kind  of  information  they  will  accept  but  also  in  their  value 
structures;  in  at  least  some  cases  where  group  decision  processes  are  used 
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It  is  important  Co  make  clear  that  these  comments  on  other  applications  are, 
at  this  point,  to  a degree  speculative,  and  are  offered  to  suggest  possible 
areas  for  further  consideration. 


THEORETICAL  CONSIDERATIONS 


The  main  purpose  of  this  paper  has  been  to  present 
procurement  process,  and  the  discussion  here  will  b 
dication  of  related  theoretical  areas.  The  origins 
of  the  process  was  largely  etapirical,  and  the  exten 
entered  into  the  process  can  be  traced  through  refe 
written  just  prior  to  that  event  and  which  was  used 
the  process  vras  based.  A more  generally  available, 
presented,  subsequently,  at  a technical  conference. 
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Any  reasonably  thorough  examination  of  the  academic  and  philistine  literature 
would  uncover  a virtually  unlimited  variety  of  articles  related  to  an  area  as 
complex  and  pervasive  as  this  one.  No  attempt  has  been  made  to 
purposes  of  this  paper,  but  an  attempt  will  be  uiade  to  identify 
major  areas  and  to  include  some  representative  references  which  may  provide  a 
useful  frame  of  reference.  First,  and  perhaps  most  obvious,  is  the  extensive 
area  of  procurement,  including  federal  and  armed  services  procurement  regula- 
tions, their  statutory'  bases,  amplifying  and  clarifying  administrative  and 
court  decisions,  and  a wide  variety  of  related  agency  policies  and  procedures. 
Complementing  this  government  documentation  are  articles  in  trade  magazines, 
books,  conference  proceedings,  scientific  journals,  and  law  reviews  commenting 
on  present  practice  and/or  suggesting  alternatives  and  improvements . The 
importance  of  procurement  to  industry  has  generated  a market-oriented  "how  to" 
literature  which  may  be  the  most  accurate  assessment  of  present  procurement 
practice,  and  there  is  a related  literature  on  project  and  systems  management. 
Less  directly  related  to  procurement  but  concerned  more  directly  with  the 
technical  and  related  problems  in  the  design  and  management  of  systems,  and 
comparably  complex  problems,  is  the  relatively  diffuse  literature  on  systems 
engineering,  systems  management,  systems  analysis,  cost-benefit  analysis, 
urban  planning,  and  specific  kinds  of  systems,  e.g.,  transportation,  health 
care,  environmental. 


In  the  scientific  literature  there  is  an  early  and  extensive  record  of  work  in 
small  group  theory  which  is,  perhaps,  as  good  a place  to  start  as  any  other 
comparably  specific  area  in  the  wide  variety  of  research  in  the  management, 
management  science,  organization  theory,  behavioral  science,  etc.,  literature. 
Simon's  early  work  [5]  is  representative  of  a small  and  select  group  of 
authors  who  described  the  administrative  decision  process  in  terms  which  were 
clear  and  understandable.  Several  different  "schools"  have  developed  to  pur- 
sue the  problems  of  decision  making,  some  concentrating  on  empirically-based 
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behavioral  models  and  others  on  more  formal  analytical  models,  with  or  without 
empirical  testing.  In  this  latter  group,  and  of  central  importance  to  the 
present  subject,  are  a number  of  researchers  variously  described  as  decision 
or  utility  theorists.  With  some  risk  of  simplification,  reference  will  be 
made  to  a useful,  and  brief  collection  of  readings  [3]  which  includes  an  arti- 
cle more  or  less  directly  applicable,  entitled  "On  Subjectively  Optimum  Se- 
lections among  Hulti-attribute  Altemat  ives , " [4  ] It  should  be  made  clear  that 
the  exploration  of  the  theoretical  implications  of  the  proposed  process,  and 
the  examination  and  further  testing  of  the  specific  mechanisms  used,  remains 
to  be  done. 

One  other,  and  extensive,  literature  is  relevant  to  the  present  paper,  and 
that  is  the  subject  of  program  evaluation.  Two  basic  references  are 
Sucteian  [6]  and  Caro  [2];  and  two  useful  and  relevant  current  discussions 
appear  in  Federal  Evaluation  Policy  [16]  and  Academic  and  Entrepreneurial  Re- 
search. [1*] 

The  subject  area  has  been  a toatter  of  interest  to  the  present  writer,  starting 
with  the  previously  mentioned  papers.  [8]  [10]  The  general  area  of  systems  is 
represented  in  two  papers  [12]  [13];  the  problems  of  cooperative  decision 
making  are  discussed  in  two  papers  [7]  [9];  and  the  subject  of  evaluation  is 
described  in  another.  [11]  These  do  not  provide  the  comprehensive  basis  that 
a more  thorough  review  of  the  literature  would  achieve,  but  they  do  provide  an 
additional  perspective  on  the  considerations  which  were  instrumental  in  the 
present  application. 


SUMMARY 

A brief  description  has  been  presented  of  a procurement  process  developed  and 
applied  to  meet  a specific  procurement  problem.  To  provide  a modest  perspec- 
tive, additional  comments  have  been  made  on  background,  the  salient  character- 
istics and  applications  of  the  process,  and  relevant  theory. 

While  the  process  has  been  previously  "tested,"  the  current  application  is  in 
mid'process,  and  the  "test"  remains  to  be  completed.  From  past  experience, 
the  process  should  be  successful  in  rating  bidders  from  strong  to  weak,  and 
in  choosing  a "winner"  whose  performance  after  award  will  reasonably  meet  our 
expectations. 
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TABLE  I PROPOSAL  EVALUATION  FACTORS 


I.  Management 

This  factor  is  a measure  of  the  background  and  overall  management  capa- 
bility of  Che  bidder. 

II.  Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's  ability  to  assure  chat  the 
Government  is  able  to  effectively  and  progressively  review  a very  large 
scale  effort  with  a minimum  of  people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  with  those  of  other  related  individuals  and 
organizations. 

III.  Objectives 

This  factor  is  a measure  of  the  bidder's  overall  understanding  of  the 
objectives  of  this  procurem.ent  (and  related  programs). 

IV.  Background  Information 

This  factor  is  a measure  of  the  bidder's  overall  understanding  of  the 
background  and  characteristics  of  this  procurement. 

V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder's  capabilities  to  carry  out  the 
evaluation  design  requirements. 

VI.  Data  Collection  Process 

This  factor  is  a measure  of  the  bidder's  capability  to  carry  out  all  o 
the  funct ions -necessary  to  the  acquisition  of  data  on  the  variables  and 
parameters  of  interest. 

VII,  Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capability  to  carry  out  all  of 
the  functions  necessary  to  the  analysis  of  the  data  and  to  present  the 
results  and  recommendations. 
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TABLE  II  FACTOR  WITH  SUB-FACTORS 
V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder's  capabilities  to  carry  out  the 
evaluation  design  requirements. 

V.l  Exploratory  (Descriptive)  and  A Priori  Proposition  Testing  Hypotheses, 
Variables  and  Parameters 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  define  the  de- 
tailed evaluation  objectives. 

V.2  Experimental  and/or  Study  Designs 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  define  the 
overall  experimental  and/or  study  design. 

V.3  Data  Collection  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  design  the 
overall  data  collection  process,  including  sources  of  information, 
sampling  strategies,  timing,  etc. 

V.4  Data  Analysis  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  design  the 
overall  data  analysis  process,  Including  processing  and  analysis,  and 
presenting  results  and  recomnendat ions . 

V,5  Phase  Two  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  redefine  the 
detailed  experimental  and/or  study  design  to  reflect  the  results  of 
Phase  One,  the  several  objectives,  and  model  of  the  process  in  a com- 
prehensive detailed  design  to  meet  both  basic  data  (base  line  data)  and 
special  data  requirements, 

V.6  Phase  Three  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  further  redefine 
the  above,  to  provide  for  a detailed  design,  and  necessary  supporting 
materials,  which  would  allow  an  on-going  evaluation  process. 
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TABLE  III  SUB-FACTOR  WITH  ILLUSTRATIVE  QUESTIONS 

V.2  Experimental  and/or  Study  Designs 

This  sub-factor  is  a measure  of  the  bidder's  ability  to  define  the  over- 
all experimental  and/or  study  design. 

V.2.1  Discuss  the  problera(s)  of  and  approaches  to  experimental  and/or  study 
design.  Alternatively,  you  coay  wish  to  demonstrate  your  capability  by  refer- 
ence to  the  relevant  education  and  experience  of  your  staff.  It  is  sug2ested 
that  you  may  wish  to  distinguish  among  the  several  kinds'  of  designs  which  may 
be  appropriate  for  meeting  the  several  kinds  of  objectives  which  are  the  out- 
come of  the  process  discussed  in  V.l. 

V.2. 2 Discuss  the  problem(s)  of  and  approaches  to  identifying  and  controlling 
for  the  potential  effects  of  parameters. 

V.2. 3 It  is  proposed  that  the  bidder  can  design  an  overall  evaluation  system 
and,»in  parallel,  carry  out  evaluations  of  specific  selected  experiments  during 
Phase  One.  Discuss  the  advantages  and  disadvantages  of  this.  Can  you  identify 
specific  experiments  which  can  be  evaluated  during  Phase  One?  Can  you  identify 
base-line  data  which  can,  or  should,  be  collected  during  Phase  One?  which  can, 
or  should,  be  collected  during  later  phases? 

V.2.4  To  what  extent  do  you  see  differences  among  the  several  PAA  which  v/ill 
affect  the  design  of  the  evaluation  system?  Will  differences  in  the  progress 
in  initiating  experiments  among  the  several  PAA  be  an  advantage  or  disad- 
vantage ? 

V.2, 5 It  is  anticipated  that  some  part  or  all  of  the  bidder's  activity  in 
this  area  will  be  dependent  upon  the  activities,  of  another  ("agency  impact") 
contractor.  How  critical  is  this  from  a time  point  of  view?  from  an  interface 
point  of  view?  Will  segmentation  (by  industry,  technology,  market)  provide 
advantages  or  disadvantages?  to  what  extent  will  longitudinal  designs  be  im- 
portant, and  will  this  be  an  advantage  or  disadvantage  (COMMERCIAL  IMPACT  ONLY) 
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APPENDIX  C 


Instructions  Given  to  Evaluation  Panel  ?>!err.be 
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RFP  6-35756 


PROPOSAL  EVALUATION  PROCEDURE 


GENERAL 

Information  and  guidance  on  proposal  evaluation  is  contained  in 
the  Proposal  Preparation  Procedure  and  Scope  of  Effort  documents. 
Covered  below  are  additional  suggestions  to  assure  a uniform, 
thorough  and  efficient  evaluation.  Comments  on  how  to  improve 
the  procedures  are  welcomed  and  should  be  furnished  to  the 
COTR  for  early  consid-eration  and  dissemination. 

INDIVIDUAL  PREPARATION 

All  proposal  evaluation  team  members  should  become  familiar 
with  the  entire  RFP,  at  least  sufficiently  to  know  what 
documents  are  contained  in  it.  All  team  members  should  have 
copies  of  the  Statement  of  Work,  Schedule (s).  Proposal 
Preparation  Procedure,  Proposal  Evaluation  Factors,  and  Scope 
of  Effort.  In  addition,  team  members  should  have  blank  copies 
of  the  Factor  (and  Sub-factor)  V7orksheets . During  the  entire 
evaluation  period,  you  should  make  notes  on  important  points 
or  questions  you  have,  and  be  prepared  to  progressively  rate 
each  bidder  on  each  sub-factor  you  have  comments  on.  These 
notes  will  aid  you  in  the  factor  team  discussions  when  the 
relative  positions  of  the  bidders  are  established. 

If  you  have  any  question  at  any  time  concerning  the  process, 
you  should  contact  the  COTR. 

• PROPOSALS 

All  copies  of  the  bidders'  proposals  will  be  received  by  the 
Contracting  Officer  on  April  15.  The  Contracting  Officer 
will  retain  the  contractual  provisions,  including  price  data, 
and  one  or  more  copies  of  the  rest  of  the  proposal.  Fifteen 
copies  of  each  bidder's  proposal,  which  includes  the  informa- 
tion responsive  to  the  Proposal  Evaluation  Factors  and  Scope 
of  Effort  documents,  will  be  provided  to  the  COTR.  The  COTR 
will  provide  copies  to  the  members  of  the  evaluation  team, 
together  with  any  required  additional  guidance  or  forms.  A 
sign-out  procedure  will  be  used  to  keep  track  of  the  proposals. 
Access  will  be  limited  to  members  of  the  evaluation  team,  and 
to  personnel  of  DOC  whose  official  responsibilities  require 
access . 

PRELIMINARY  REVIEW 

It  is  proposed  that  all  team  members  will  spend  Friday, 

April  16,  at  ETIP  to  accomplish  a preliminary  review.  Proposals 
will  be  distributed  in  the  morning,  and  members  will  spend 
most  of  the  day  reviewing  the  proposals  both  overall  and  with 
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respect  to  specific  factors  and  sub-factors.  The  purpose  of 
this  is  the  follov;ing:  a)  to  estimate  the  overall  magnitude 
of  time  required  for  reviev;  as  a basis  for  planning  and 
scheduling  the  reviev;  process;  b)  to  identify  any  critical 
problems  faced  by  evaluation  team  members;  c)  to  identify 
any  critical  specific  questions  v/hich  must  be  directed  to  a 
specific  bidder  (through  the  C.O.);  d)  to  determine  if  one 
or  more  factor  teams  will  be  ready  to  meet  by  .'-'onday  .morning; 
e)  to  establish  which  of  the  tv;o  procurements  will  be  evaluat 
first;  and  f)  to  review  the  assignments  of  team  me.mbers  to 
specific  sub-factors. 

By  the  end  of  the  day  (Friday) , it  is  proposed  to  schedule  on 
or  more  factor  team  meetings  on  ‘lo.nday;  a.nd , possibly  te.ntati 
schedule  subsequent  factor  team  meetings. 

QUESTIONS  r 


Because  it  is  not  anticipated  t.hat  any  cf  the  individual  bidd 
will  be  invited  for  oral  presentations  and  questioning  (prior 
to  the  completion  of  the  factor  tea.m  m.eemings;  , a.ny  additiona 
information  from  the  bidders  can  be  obtained  only  by  directi.n 
specific  questions  to  the  specific  bidder  through  the  Contrac 
Officer.  These  questions  should  be  identified  by  sub-factor, 
and  limited  to  the  clarification  of  ambiguities  or  to 
clearing  up  omissions  (on  apparent  strong  or  v;eak  points,  v;hi 
would  be  expected  to  substantially  affect  the  team's  ability 
to  rate  the  sub-factor.  ^ im.por tant  that  these  be  ident i 

f ied  as  early  as  possible  to  allcw  time  to  obtain  a.nswers  ; 
waiting  for  an  answer  m.ay  delay  the  scheduling  of  the  af f ecte 
factor  team  m.eetina. 

- - - I , , , 

Specific  questions  should  be  entered  on  the  Question  Form,  and 
given  to  the  COTR.  Because  of  the  time  delay  involved,  each 
question  should  be  reviewed  to  be  sure  that  the  additional 
information  is  absolutely  necessary. 

If  additional  information  is  required  from-  someone  other  than 
the  bidder  (e.g.,  a COTR  on  another  program),  put  that  quest! 
on  a Question  Form,  and  give  to  the  COTR. 

Note:  No  questions  should  be  directed  to  anyone  outside  of 

the  evaluation  team,  except  by  the  COTR  through  the  CO. 

INDIVIDUAL  REVIEW 

The  process  of  individual  review  is  largely  within  the  discre 
of  the  evaluation  team  member.  There  are  only  two  specific 
checkpoints:  1)  that  you  identify  specific  questions  (see 
previous  heading)  as  early  as  possible,  and  2)  that  you  have 
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completed  your  review  of  the  relevant  (and/or  assigned)  part 
of  the  bidders'  proposals  prior  to  each  factor  team  meeting 
(which  will  be  scheduled  with  this  requirement  in  mind) . 

All  evaluation  team  members  are  encouraged  to  review  all 
parts  of  all  proposals,  and  may  attend  all  factor  team 
meetings . 

It  is  recommended  that  all  members  of  a specific  factor  team 
review  all  of  the  sub-factors  which  make  up  the  factor. 

Each  member  of  a factor  team  must  review  the  sub-factors  to 
which  he  is  assigned.  For  this  purpose  (and,  optionally, 
for  the  others),  a Factor  (and  Sub-factor)  Worksheet  must  be 
completed  for  each  assigned  sub-factor.  You  may  use  duplicate 
copies  of  the  Worksheet,  or  any  convenient  form.  The  purpose 
is  to  be  sure  that  you  have  established  clearly  what  the 
bidder  proposes,  and  that  you  have  a quick  reference  to 
your  rating  and  supporting  reasons  for  them  for  use  during 
the  factor  team  meeting. 

FACTOR  TEAxM  MEETINGS 

Members  of  the  proposal  evaluation  team  who  have  been  assigned 
to  one  or  more  sub-factors  within  a specific  factor  make  up 
the  "factor  team"  for  that  factor. 

When  the  members  of  the  team  are  ready,  a factor  team  meeting 
will  be  scheduled.  These  factor  team  meetings  will  be 
scheduled  separately  for  each  factor  and  will  last  about 
an  hour. 

The  procedure  of  the  factor  team  meetings  will  include  the 
following  steps: 

First,  an  informal  discussion  and  comparing  of  notes  to 
establish  that  we  are  using  the  same  standards  and  information 
(to  uncover  major  discrepancies  or  omissions) . 

Second,  comparing  and/or  discussing  individual  reviews,  and 
rough  ranking.  • 

Third,  establishing  relative  differences  or  distribution. 

The  following  ground  rules  should  be  observed  in  the  conduct 
of  the  factor  team  meetings: 

Information  to  be  considered: 

1st  - RFP,  and  associated  documents 

2nd  - Written  proposals,  including  "Scope  of  Work" 
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3rd  - Responses  to  specific  questions,  if  any 

4th  - Individual  experience  of  the  factor  team  members  - 

if  there  is  critical  personal  information,  it  should 
be  presented  to  the  team  for  consideration . 

Rating  (factor  and  sub-factor)  shall  be  by  considering  these 
points,  all  together: 

1st  - Comprehension  or  understanding  of  the  problem 
presented . 

2nd  - Approach  described  or  outlined 

3rd  - Present  or  potential  capability  to  accomplish  this. 

(Rating  of  potential  capability  is  a very  importan*: 
point-the  teams  must  try  to  project,  formally,  the 
‘^capability  of  the  bidder  to  correct  def icie.ncies 
which  are  due  to  misunders ta.nding  or  inadver fence . ) 

Major  eiT'phasis  should  be  on  the  factor  or  sub-factor  bei.nc 
rated;  however,  no  factor  exists  i.n  total  isolation,  a.nd  the 
impact  of  other  factors  should  not  be  excluded.  The  overall 
standard  is  "the  extent  to  v.'hich  you  believe  the  factor  will  be 
accomplished,  keeping  in  mind  his  unders tandi.ng , approach  a.nd 
capability . " 


Comparative  "weighting"  of  sub-factors  and  i.ndividual  evalua- 
tions: The  factor  team  is  expected  to  hammer  out  the  comparative 
evaluation  of  the  several  sub-f  actors ; there  is  no  preco.nceived 
arbitrary  division.  Similarly,  the  factor  team  mem.bers  are 
required  to  establish  a mutually  satisfactory  evaluation.  Other 
members  of  the  factor  team  may ’properly  i.nquire  i.nto  the  basis 
$nd  reasoning  of  a specialist  in  his  area;  a.nd,  conversely, 
should  give  appropriate  weight  to  his  evaluation.  No  arbitrary 
voting  procedure  is  established. 

The  process  to  be  used  in  arranging  the  order  and  distribution  o 
the  contractors  is  to  be  established  by  each  factor  team. 

Various  summary  steps,  such  as  proceedi.ngs  by  sub-factors,  by 
individual  evaluators,  or  the  overall  factor,  m^ay  be  used. 

Distribution:  After  a rough  ran.king  of  the  bidders,  the  tea.m 

should  separate  the  bidders  to  indicate  the  distribution,  that  i 
the  relative  gaps  reflecting  the  differences  observed.  This  m^ay 
be  accomplished  by  having  a team  miember  adjust  themi  on  a black- 
board until  the  team  agrees  on  the  distribution.  Once  a.n 
order  and  a distribution  have  been  mutually  agreed  upon,  the 
distribution  shall  be  transcribed  to  record  form,  (the  Fa 
Worksheet)  for  file  purposes  and  signed  by  each  mie.mber  o 
factor  team.  Any  strong  individual  disagreemients  wioh  t 
outcome  of  the  team  should  be  noted  on  the  record  form 
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together  with  a concise  supporting  statement.  Also,  any 
points  which  will  require  negotiating  in  the  event  a certain 
bidder  is  ultimately  chosen,  should  be  noted  on  the  same  form. 

When  using  the  blackboard  as  a graphical  scale,  these  suggestions 
are  made.  The  top  of  the  board  should  be  reserved  for  the 
bidder,  if  any,  who  makes  you  "sing,"  you  believe  that  nothing 
can  stop  him  from  being  a success  on  this  factor  (or  sub-factor) . 
The  bottom  should  be  reserved  for  the  bidder  who  is  hopeless 
within  reasonable  money  and  time  limits.  Care  should  be  taken 
to  avoid  bunching  everybody  in  the  middle.  Reflect  the  com- 
parative value  of  the  bidders  over  the  total  scale. 

REVIEW 


The  COTR  will  incorporate  the  factors  and  prepare  a summary 
of  the  evaluation  for  presentation  and/or  any  procurement 
review.  Procurement  practice  requires  that  this  stage  and 
up  to  signing  of  contract  be  limited  to  minimum  personnel 
required  to  prepare  reviev/  documents. 
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APPENDIX  D 


Ratings  from  the  Agency  Imapct  Contract  Evaluation 

Displays  of  the  way  the  ratings  appeared  on  t*^e  blackboar 
during  the  factor  team  meetings.  The  initial  individual 
ratings  and  the  agreed  upon  sub-factor  ratings  appear 
on  one  sheet  per  sub-factor. 

The  displays  of  agreement  on  factor  ratings  are  provided 
after  the  last  sub-factor  within  each  factor. 
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APPENDIX  E 


Ratings  from  the  Commercial  Imapct  Contract  Evaluation 


Displays  of  the  way  the  ratings  appeared  on  the  blackboard 
during  the  factor  team  meetings.  The  initial  individual 
ratings  and  the  agreed  upon  sub-factor  ratings  appear 
on  one  sheet  per  sub-factor. 

The  displays  of  agreement  on  factor  ratings  are  provided 
after  the  last  sub-factor  within  each  factor. 
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APPENDIX  F 


Cover 


letter  and  Questionnaire 


for  Panel  Members 


February  15,  1980 


Dear  ; 

I am  writing  to  request  your  assistance  in  a case  study  of  a proposal  evaluation 
process  in  which  you  participated  in  April  of  1976.  The  procurement  was  for  the 
development  of  evaluation  systems  for  experiments  in  government  procurement 
on  behalf  of  the  Experimental  Technology  Incentives  Program  (ETIP).  This  case 
study  is  designed  to  assess  the  effectiveness  of  the  method  which  ETIP  used  to 
solicit  and  evaluate  proposals.  Your  answers  to  the  enclosed  questions  will  provide 
ETIP  with  a basis  for  reaching  conclusions  on  several  parts  of  this  evaluation. 

The  total  time  for  your  participation  will  probably  be  between  a half  hour  to  an 
hour. 

As  you  may  recall,  one  of  the  key  characteristics  of  the  RFP  was  the  use  of  a 
factor/sub-factor  structure  which  required  responses  to  issues  which  were  thought 
to  be  critical  for  the  procurement.  To  help  you  in  answering  the  questions  I have 
enclosed  a list  which  contains  the  factor  and  sub- factor  names  and  a summary 
statement  of  each  one. 

You  need  not  mail  the  questionnaire  back.  I will  be  calling  you  in  about  a w^eek 
to  try  to  provide  any  clarifications  and  to  obtain  your  answers.  Prior  to  this,  olease 
go  through  the  questions  and  write  down  your  initial  answers.  Your  name  or  the 
name  of  the  firm  which  you  represented  will  not  be  identified  in  the  reoort  on 
this  research. 

This  case  study  will  be  published  in  NTIS  and  be  disseminated  to  people  interested 
in  the  procurement  of  evaluation  systems. 

Sincerely, 


Bud  Libman 
Policy  Analyst 

Experimental  Technology  Incentives  Program 
Center  for  Field  Methods 


Enclosures 


Background  Note 

The  RFP  to  which  these  questions  refer  was  for  the  procurement  of  two  systems 
to  evaluate  the  impacts  of  experiments  with  government  procurement  methods. 

One  system  was  to  evaluate  impacts  on  the  commercial  sector  and  the  other 
was  to  evaluate  impacts  on  governmient  agencies.  They  were  each  estimated  to 
require  200  man-months  of  effort  for  the  first  two  phases  of  development. 

It  appeared  to  ETIP  that  there  was  considerable  complexity  and  uncertainty  Involved 
in  developing  these  systems.  Complexity  came  in  part  from  requirements  to  com- 
bine the  technical  concerns  of  doing  evaluations  and  designing  the  systems  with 
the  political  concerns  of  meeting  the  information  needs  of  people  at  different 
levels  in  several  organizations.  Uncertainty  was  present  since  objectives  were 
expected  to  shift  over  the  three  year  period  of  the  contracts. 

Three  proposals  were  received  for  the  commercial  impact  contract.  Six  proposals 
were  received  for  the  agency  impact  contract. 

The  evaluation  panel  met  in  factor  teams  during  the  week  after  the  proposals 
were  received.  You  participated  on  factors  on  the  commercial  im  pact  con- 
tract evaluation  and  factors  on  the  agency  impact  contract  evaluation. 


Questions 


1.  This  question  addresses  the  effectiveness  of  the  RFP's  factor  sub-factor  struc- 
ture for  enabling  the  panel  to  evaluate  the  bidders'  qualifications. 

a)  Did  you  feel  that  the  factor/sub-f actor  structure  enabled  bidders  to  dem.on- 
strate  their  ability  to  deal  with  critical  problems  which  might  arise  during 
the  project? 

Yes  No 


If  no,  what  modifications  in  the  structure  would  have  enabled  bidders  to 
better  demonstrate  their  abilities? 


b)  Did  you  feel  that  there  were  critical  gaps  in  evaluating  the  bidders'  capabi- 
li  ties  to  perform  effectively  on  the  project? 

Yes  No 


vn 
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If  yes,  what  were  these  gaps? 


c)  Were  you  able  to  readily  locate  the  information  in  the  proposals  which 
was  relevant  to  the  issues  you  were  evaluating? 

Yes  No 


If  no,  what  changes  in  the  structure  would  have  helped  you  to  more  readily 
locate  the  information? 


2.  This  question  addresses  the  appropriateness  of  the  sub-factors  which  were 
included  in  the  RFP. 

a)  Did  you  feel  that  the  sub-factors  encompased  most  of  the  issues  which 
would  be  critical  to  designing  the  evaluation  systems  which  ETIP  desired? 

Yes  No 

If  no,  which  important  issues  were  not  adequately  covered?  


b)  Did  you  feel  that  many  of  the  sub-factors  dealt  with  issues  unimportant 
for  designing  the  evaluation  systems  which  ETIP  desired? 

Yes  No 

If  yes,  which  sub-factors  or  issues  did  you  feel  were  unimportant?  
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3.  This  question  addresses  whether  the  number  of  proposals  evaluated  was  suf- 
ficient to  enable  panel  members  to  be  confident  in  their  ratings. 

Do  you  feel  that  being  able  to  rate  more  proposals  would  have  increased  your 
confidence  in  the  accuracy  of  the  scores  which  were  assigned'’ 

a)  For  the  agency  impact  evaluation  system  contract? 

Yes  No 

If  yes,  about  how  many  more? 

b)  For  the  commercial  impact  evaluation  system  contract? 

Yes  No 

If  yes,  about  how  many  more? 


4.  This  question  addresses  the  effectiveness  and  the  equity  of  the  lack  of  restric- 
tions on  admissible  information.  Of  concern  is  the  effect  of  knowing  the 
identity  of  the  bidders  (i.e.,  non-blind  evaluation)  and  being  able  to  use  reports 
of  the  experiences  of  other  panel  members  who  had  previously  workec  w-itn 
the  bidders. 

Did  being  able  to  use  all  available  information  about  bidders  in  the  evaluation: 
a)  improve  the  accuracy  of  the  scores? 

Yes  No 


b)  introduce  unfair  bias  into  the  scores? 

Yes  No 


If  yes,  how  did  this  occur? 
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5.  What  did  you  feel  the  strongest  aspect  of  the  evaluation  process  was? 


6.  What  did  you  feel  the  weakest  aspect  of  the  evaluation  process  was? 


7.  Approximately  how  many  other  evaluation  panels  have  you  served  on? 


8.  Of  the  total  number  of  other  proposal  evaluation  panels  which  you  have  served 
on,  about  how  many  were  for  projects  of  the  magnitude  (i.e.,  200  man-months), 
complexity,  and  uncertainty  of  this  one? 


Bidders  were  furnished  the  following  information  either  in  the  Preliminary  Description 
or  at  the  first  Bidders  Conference  on  February  26,  1976: 

Information  for  Prospective  Bidders 

Corrected  Version  of  the  Commerce  Business  Daily  Announcement 
Introductory  Notes 
Definitions 
Statement  of  Work 

Statement  of  Work  (with  illustrative  sub-items  for  Phase  One) 

Proposed  Schedule  - Overall 
Schedule  Phase  One 
Scope  of  Effort 

Proposal  Preparation  Procedure  (with  revisions) 

Proposal  Evaluation  Factors 
Proposal  Evaluation  Factors  with  Illustrations 
Numerical  Weights  for  Proposal  Evaluation  Factors 
ETIP  Program  Plan,  dated  February  5,  1974 
ETIP  Progress  Report,  dated  April  4,  1975 
ETIP  Progress  Report,  dated  August  1975 

Selected  Bibliography  of  Published  Documents  Related  to  This  Procurement 
Message  from  the  President  of  the  United  States  Concerning  Science  and  Technology 
Evaluation  System  Design  Process 

Description  of  Governmental  Procurement  Policy  Program  - ETIP 
The  Council  of  State  Governments  and  Associated  Organizations 
NIGP  Dictionary  of  Purchasing  Terms 
NIGP  1974  Annual  Conference  and  Products  Exposition 

Memorandum  to  Members  Regarding  NIGP/ETIP  Project,  dated  October  9,  1975 

Memorandum  to  Members  Regarding  NIGP/ ETIP  Project  on  Window  Air  Conditioners, 
dated  November  26,  1975 

NIGP  Membership  Application 

Prerequisites  for  Certification  of  Public  Purchasing  Agents 
NIGP  Letter  Service 
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APPENDIX  G 


Cover  letter  and  Questionnaire  for  Bidder  Representatives 


February  15,  1980 


Dear  : 

I am  writing  to  request  your  assistance  in  a case  study  of  a Deoartment  of  Com- 
merce procurement  which  you  responded  to  in  Aorii  of  1976  ^RFP  6-35736).  The 
procurement  was  for  the  development  of  evaluation  systems  for  exoerimients  m 
government  procurem.ent  on  behalf  of  the  Experimental  Technology  Incentives 
Program  (ETIP).  This  case  study  is  designed  to  assess  the  effectiveness  of  tne 
method  which  ETIP  used  to  solicit  and  evaluate  orooosals.  Your  answers  to  the 
enclosed  questions  will  provide  ETIP  with  a basis  for  reaching  conclusions  on  several 
parts  of  this  evaluation.  The  total  time  for  your  participation  will  probably  be 
between  a half  hour  to  an  hour. 

As  you  may  recall,  one  of  the  key  characteristics  of  the  RFP  was  the  use  of  a 
factor/sub-factor  structure  which  required  responses  to  issues  which  were  thought 
to  be  critical  for  the  procurement.  To  help  you  in  answering  the  auestions  I have 
enclosed  a list  which  contains  the  factor  and  sub-factor  names  and  a summary 
statement  of  each  one.  I have  also  enclosed  a list  of  the  written  materials  which 
were  included  with  the  RFP. 

You  need  not  mail  the  questionnaire  back.  I will  be  calling  you  in  about  a week 
to  try  to  provide  any  clarifications  and  to  obtain  your  answers.  Prior  to  this,  please 
go  through  the  questions  and  write  down  your  initial  answers.  Your  name  or  the 
name  of  the  firm  which  you  represented  will  not  be  identified  in  the  report  of 
this  research. 

This  case  study  will  be  published  in  NTIS  and  be  disseminated  to  people  interested 
in  the  procurement  of  evaluation  systems. 

Sincerely, 


Bud  Libman 
Policy  Analyst 

Experimental-  Technology  Incentives  Program 
Center  for  Field  Methods 


Enclosures 
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Background  Note 

The  RFP  to  which  these  questions  refer  was  for  the  procurement  of  two  systems 
to  evaluate  the  impacts  of  experiments  with  government  procurement  methods. 

One  system  was  to  evaluate  impacts  on  the  commercial  sector  and  the  other  was 
to  evaluate  impacts  on  government  agencies.  They  were  each  estimated  to  require 
200  man-months  of  effort  for  the  first  two  phases  of 'development. 

It  appeared  to  ETIP  that  there  was  considerable  complexity  and  uncertainty  involved 
in  developing  these  systems.  Complexity  came  in  part  from  requirements  to  com- 
bine the  technical  concerns  of  doing  evaluations  and  designing  the  systems  with 
the  political  concerns  of  meeting  the  information  needs  of  people  at  different 
levels  in  several  organizations.  Uncertainty  was  present  since  objectives  were 
expected  to  shift  over  the  three  year  period  of  the  contracts. 


Questions 


1.  This  question  addresses  the  effectiveness  of  the  RFP's  factor/sub-factor  structure 
for  enabling  bidders  to  demonstrate  their  qualifications. 

a)  Did  you  feel  that  the  factor/sub-factor  structure  enabled  bidders  to  demon- 
strate their  ability  to  deal  with  the  problems  which  might  arise  during 
the  project? 

Yes  No 


If  no,  what  modifications  in  the  structure  would  have  enabled  bidders  to 
better  demonstrate  their  abilities? 


b)  Did  following  the  factor/sub-factor  structure  create  any  problems  for 
your  team,  in  trying  to  demonstrate  its  capability,  which  would  not  have 
been  present  if  a different  structure  had  been  used? 

Yes  No 

If  yes,  what  special  problems  were  created  by  the  sub-factor  structure? 
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c)  Was  your  team  in  any  manner  placed  at  a competitive  disadvantage  tiecause 
you  had  to  follow  the  factor/sub-factor  structure'’ 

Yes  No 

If  yes,  please  describe  how  you  were  disadvantaged.  


2.  This  question  addresses  whether  the  proposal  format  enaoled  bidcers  to  be 
confident  that: 

(1)  they  included  information  in  their  proposals  on  all  of  the  capabilit.es 
which  the  evaluation  panel  would  be  looking  for,  and 

(2)  the  panel  would  be  able  to  find  this  information  when  it  was  reeded. 

a)  Were  you  confident  that  you  had  written  something  to  demonstrate  your 
qualifications  on  all  of  the  capabilities  which  the  panel  a'ouIc  oe  evahjat.r.g'' 


Yes 


No 


If  no,  why  not? 


b)  Were  you  confident  that  the  panel  would  find  the  information  in  your  pro- 
posal which  was  relevant  to  each  specific  capability  at  the  time  when 
that  capability  was  being  evaluated? 

Yes  No 


If  no,  what  information  did  you  feel  might  be  overlooked  and  why  did  you 
feel  that  this  might  happen? 
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3.  This  question  addresses  whether  the  information  included  in  the  solicitation 
document,  the  supporting  documents  and  the  two  briefings  gave  bidders  a 
sufficient  general  understanding  of  what  ETIP  wanted  to  buy.  Since  many 
of  the  specific  objectives  of  the  projects  were  not  known  when  the  RFP  was 
issued  the  concern  here  is  whether  sufficient  information  was  provided  on 
the  general  nature  of  the  projects  to  enable  bidders  to  respond  effectively 
to  the  sub-factor  issues. 

a)  Did  the  information  provided  by  ETIP  allow  you  to  assess  the  skills  which 
would  be  required  to  do  the  work? 

Yes  No 


If  no,  what  additional  information  would  have  helped? 


b)  Was  ample  information  provided  in  the  briefings  and  supporting  documents 
so  that  you  were  confident  that  you  had  sufficiently  related  the  sub-factor 
issues  to  the  objectives  of  the  projects? 

Yes  No 


If  no,  what  type  of  additional  information  would  have  been  useful? 


4.  This  question  addresses  the  appropriateness  of  the  sub-factors  which  were 
included  in  the  RFP. 

a)  Did  you  feel  that  the  sub-factors  encompased  most  of  the  issues  which 
would  be  critical  to  designing  the  evaluation  systems  which  ETIP  desired? 


Yes 


No 


If  no,  which  important  issues  were  not  adequately  covered*’ 


b)  Did  you  feel  that  many  of  the  sub-factors  dealt  with  ;ss'.es  jr.irr.portar.t 
for  designing  the  evaluation  system’s  which  ETIP  cesirec*’ 

Yes  No 

If  yes,  which  sub-factors  or  issues  did  you  feel  *.ere  jram portar.t*’  


5.  This  question  adcresses  the  degree  to  which  bidders  jnderstood  tr.e  proposal 
evaluation  process  which  ETIP  used. 

a)  Did  the  information  provided  enable  you  to  uncerstanc  tre  process  wncn 
ETIP  would  use  to  assign  scores  on  each  sub-factor? 

Yes  No 

If  no,  please  describe  which  aspects  of  the  evaluation  process  were  not 
clear.  


b)  Did  the  information  provided  enable  you  understanc  how  tr.e  ratings  on 
the  various  sub-factors  would  be  combined  to  arrive  at  fl.nai  scores? 

Yes  No 

If  no,  what  aspects  of  this  were  not  clear? 
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6.  This  question  addresses  whether  the  process  of  combining  scores  appeared 
equitable.  The  RFP  provided  weights  for  combining  the  factor  scores  into 
a final  score.  However,  the  relative  weightings  of  the  sub-factors  within 
the  factors  was  left  to  the  discretion  of  each  factor  team.  Were  you  concerned 
that  not  specifying  the  relative  weightings  of  the  sub-factors  might  lead  to 
inequities  in  the  process? 

Yes  No 


If  yes,  how  so? 


7.  Did  you  feel  that  the  process  would  be  a) difficult  or  b) easy  to  cor- 

rupt? i.e.,  For  one  person,  or  a group  of  people,  who  favored  a specific  bidder 
to  exercise  sufficient  control  over  the  process  to  sway  the  award  decision. 
Please  explain  your  answer.  


8.  Approximately  how  many  other  RFPs  have  you  responded  to? 


9.  Of  the  total  number  of  other  RFPs  which  you  have  responded  to,  about  how 
many  were  for  projects  of  similar  size  (i.e.,  200  man-months),  complexity, 
and  uncertainty  as  this  one? 


10.  What  do  you  feel  was  the  strongest  aspect  of  the  process  which  ETIP  used? 


11.  What  do  you  feel  was  the  weakest  aspect  of  the  process  which  ETIP  used  ? 
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APPENDIX  H 


Enlargement  of  Figure  IV.  1 


The  Activities  in  ETIP's  Source  Selection 
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